{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca50d443-4bb4-429b-90fd-c8fa2b4d5852",
   "metadata": {},
   "source": [
    "- https://jfan001.medium.com/how-we-cut-the-rate-of-gpt-hallucinations-from-20-to-less-than-2-f3bfcc10e4ec\n",
    "- https://cookbook.openai.com/examples/using_logprobs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c0cbb3f-b3af-4a81-bc57-a888f2a85e86",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{split}\n",
    "&\\text{logp}=\\log(p)\\\\\n",
    "&p=\\exp(\\log(p))\n",
    "\\end{split}\n",
    "$$\n",
    "\n",
    "- 定量的感知模型 API 的输出；\n",
    "- `logprobs`: the API returns the log probabilities of **each output** token, along with a limited number of the most likely tokens at each token position and their log probabilities.\n",
    "    - `logprobs`: Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message. This option is currently not available on the gpt-4-vision-preview model.\n",
    "    - `top_logprobs`: An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.\n",
    "- 数值分析 & 应用\n",
    "    - $0 \\leq p\\leq 1 \\rightarrow \\log (p)\\leq 0$\n",
    "    - Logprob can be any negative number or 0.0.\n",
    "        - 0.0 corresponds to 100% probability.\n",
    "        - check if the `p=e^(logprob)` of the true token is below 90%. \n",
    "    - Logprobs allow us to compute the **joint probability** of a sequence as the sum of the logprobs of the individual tokens. This is useful for scoring and ranking model outputs. Another common approach is to take the average per-token logprob of a sentence to choose the best generation.\n",
    "        - $\\log P(S)=\\log P(x_1,x_2,...,x_n)=\\sum_i \\log (x_i|x_{\\lt x_{i-1}})$\n",
    "        - $\\frac1n\\sum_i\\log P(x_i|x_{<i})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "18c4f088-7b2c-4eb9-ad96-ec2c04d9568a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:48:39.819807Z",
     "iopub.status.busy": "2024-12-07T11:48:39.819184Z",
     "iopub.status.idle": "2024-12-07T11:48:40.627833Z",
     "shell.execute_reply": "2024-12-07T11:48:40.626525Z",
     "shell.execute_reply.started": "2024-12-07T11:48:39.819758Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAroAAAHWCAYAAACYIyqlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABIUUlEQVR4nO3deXiU1f3+8XuyTQgQICQQkAgk7IKAeIEREKSyiArUVlpXUlyqgvoTqsUvVIiKglKlWquoCFq02kXcoEgUtCqLla0CEUGDrGGLECCQTDLn9wdmJGRhJnlme+b9ui4uM2fOzHzCIe3N4fOcx2GMMQIAAABsJirYBQAAAAD+QNAFAACALRF0AQAAYEsEXQAAANgSQRcAAAC2RNAFAACALRF0AQAAYEsEXQAAANgSQRcAAAC2RNAFAJvYvn27HA6HZs2aZdl7Tps2TQ6HQwcPHjzr3DZt2igrK8vz+OOPP5bD4dDHH3/sGcvKylKbNm18+mwAqC2CLgBbmj9/vhwOh7788stgl+IJbOW/EhIS1KVLF02ZMkWFhYXBLi+oioqKNG3atAphGACsEhPsAgAgUjz33HNq0KCBjh07pqVLl2r69OlatmyZPv/8c1vsXG7ZskVRUTXvn7z44otyu92ex0VFRcrOzpYkDRw4sMLcKVOmaNKkSZbXCSByEHQBIEB++ctfKjk5WZJ0++236xe/+IXeeustrVq1SpmZmVW+pqioSAkJCYEss9acTudZ58TGxnr9fjExMYqJ4f+mANQerQsAItq6det0+eWXKzExUQ0aNNDPfvYzrVq1qtK8//3vfxowYIDq1aunVq1a6ZFHHtG8efPkcDi0ffv2Wn32oEGDJEl5eXmSTu1odu3aVWvWrNEll1yihIQE/d///Z8kaf/+/br55pvVvHlzxcfHq3v37nrllVeqfe+nnnpKrVu3Vr169TRgwABt3Lix0veTlZWl9PR0xcfHKzU1VWPHjtWhQ4eqfL+DBw9q9OjRSkxMVNOmTXXPPffo5MmTFeac2aNbldN7dLdv366UlBRJUnZ2tqe1Y9q0aZKq79FdsGCBevXqpXr16ikpKUm//vWvtXPnzgpztm7dql/84hdKTU1VfHy8WrVqpV//+tc6cuRIjfUBsBf+qgwgYm3atEn9+/dXYmKi7r//fsXGxmrOnDkaOHCgPvnkE/Xp00eStHv3bl166aVyOBx64IEHVL9+fb300kte7WDW5Ntvv5UkNW3a1DN26NAhXX755fr1r3+tG264Qc2bN9eJEyc0cOBAbdu2TePHj1fbtm31j3/8Q1lZWTp8+LDuueeeCu/76quv6ujRoxo3bpxOnjypP/3pTxo0aJC++uorNW/eXJKUk5Oj7777Tr/5zW+UmpqqTZs26YUXXtCmTZu0atWqSgFz9OjRatOmjR577DGtWrVKTz/9tH744Qe9+uqrtf7+U1JS9Nxzz+mOO+7Qz3/+c1199dWSpPPPP7/a10yfPl1/+MMfNHr0aN1yyy06cOCAnnnmGV1yySVat26dGjdurJKSEg0dOlTFxcW66667lJqaqt27d+v999/X4cOH1ahRo1rXDCDMGACwoXnz5hlJ5r///W+1c0aNGmXi4uLMt99+6xnbs2ePadiwobnkkks8Y3fddZdxOBxm3bp1nrFDhw6ZpKQkI8nk5eXVWMvUqVONJLNlyxZz4MABk5eXZ+bMmWOcTqdp3ry5OX78uDHGmAEDBhhJ5vnnn6/w+tmzZxtJZsGCBZ6xkpISk5mZaRo0aGAKCwuNMcbk5eUZSaZevXpm165dnrmrV682ksy9997rGSsqKqpU59/+9jcjyfznP/+pVPuIESMqzL3zzjuNJLNhwwbPWOvWrc2YMWM8j5cvX24kmeXLl3vGxowZY1q3bu15fODAASPJTJ06tdrft3Lbt2830dHRZvr06RXmffXVVyYmJsYzvm7dOiPJ/OMf/6j0ngAiC60LACJSWVmZli5dqlGjRik9Pd0z3qJFC1133XX67LPPPCciLFmyRJmZmerRo4dnXlJSkq6//nqfPrNjx45KSUlR27Zt9dvf/lbt2rXTokWLKvTgOp1O/eY3v6nwusWLFys1NVXXXnutZyw2NlZ33323jh07pk8++aTC/FGjRumcc87xPO7du7f69OmjxYsXe8bq1avn+frkyZM6ePCgLrroIknS2rVrK9U+bty4Co/vuusuT22B8tZbb8ntdmv06NE6ePCg51dqaqrat2+v5cuXS5Jnx/aDDz5QUVFRwOoDEHpoXQAQkQ4cOKCioiJ17Nix0nOdO3eW2+3Wzp07dd555+n777+v8mKxdu3a+fSZ//rXv5SYmKjY2Fi1atVKGRkZleacc845iouLqzD2/fffq3379pVONOjcubPn+dO1b9++0vt26NBBf//73z2PCwoKlJ2drTfeeEP79++vMLeqPtYz3zMjI0NRUVG17k+uja1bt8oYU+X3J/10oVvbtm01YcIEPfnkk3rttdfUv39/jRgxQjfccANtC0CEIegCQIBccsklnlMXqnP6Tqs/jR49WitWrNB9992nHj16qEGDBnK73Ro2bFiF47+qE4zj0NxutxwOh/79738rOjq60vMNGjTwfP3HP/5RWVlZeuedd7R06VLdfffdnv7iVq1aBbJsAEFE0AUQkVJSUpSQkKAtW7ZUeu7rr79WVFSU0tLSJEmtW7fWtm3bKs2raswfWrdurf/9739yu90VdnW//vprz/On27p1a6X3+OabbzynHfzwww/66KOPlJ2drQcffLDG153+XNu2bT2Pt23bJrfb7fVdzqrjS2DOyMiQMUZt27ZVhw4dzjq/W7du6tatm6ZMmaIVK1aob9++ev755/XII4/UpWQAYYQeXQARKTo6WkOGDNE777xT4Z/f9+3bp9dff139+vVTYmKiJGno0KFauXKl1q9f75lXUFCg1157LSC1Dh8+XPn5+XrzzTc9Y6WlpXrmmWfUoEEDDRgwoML8t99+W7t37/Y8/uKLL7R69WpdfvnlkuTZDTXGVHjd7Nmzq63h2WefrfD4mWeekSTPe9ZWeX/y4cOHzzr36quvVnR0tLKzsyvVbozxHI1WWFio0tLSCs9369ZNUVFRKi4urlO9AMILO7oAbO3ll1/WkiVLKo3fc889euSRR5STk6N+/frpzjvvVExMjObMmaPi4mI9/vjjnrn333+/FixYoMGDB+uuu+7yHC927rnnqqCgwO//jH/bbbdpzpw5ysrK0po1a9SmTRv985//1Oeff67Zs2erYcOGFea3a9dO/fr10x133KHi4mLNnj1bTZs21f333y9JSkxM1CWXXKLHH39cLpdL55xzjpYuXeo5z7cqeXl5GjFihIYNG6aVK1dqwYIFuu6669S9e/c6fW/16tVTly5d9Oabb6pDhw5KSkpS165d1bVr10pzMzIy9Mgjj+iBBx7Q9u3bNWrUKDVs2FB5eXlauHChbrvtNv3ud7/TsmXLNH78eF1zzTXq0KGDSktL9de//lXR0dH6xS9+Uad6AYSZoJ75AAB+Un68WHW/du7caYwxZu3atWbo0KGmQYMGJiEhwVx66aVmxYoVld5v3bp1pn///sbpdJpWrVqZxx57zDz99NNGksnPz6+xlvJjsg4cOFDjvAEDBpjzzjuvyuf27dtnfvOb35jk5GQTFxdnunXrZubNm1dhTvnxYk888YT54x//aNLS0ozT6TT9+/evcAyYMcbs2rXL/PznPzeNGzc2jRo1Mtdcc43Zs2dPpaO+ymvfvHmz+eUvf2kaNmxomjRpYsaPH29OnDhR4T1rc7yYMcasWLHC9OrVy8TFxVX4/DOPFyv3r3/9y/Tr18/Ur1/f1K9f33Tq1MmMGzfObNmyxRhjzHfffWfGjh1rMjIyTHx8vElKSjKXXnqp+fDDD6v8vQVgXw5jzvj3HwCAV/7f//t/mjNnjo4dO1blxVEAgOCiRxcAvHDixIkKjw8dOqS//vWv6tevHyEXAEIUPboA4IXMzEwNHDhQnTt31r59+zR37lwVFhbqD3/4Q7BLAwBUg6ALAF4YPny4/vnPf+qFF16Qw+HQBRdcoLlz5+qSSy4JdmkAgGrQowsAAABbokcXAAAAtkTQBQAAgC3Ro3sGt9utPXv2qGHDhkG5lzsAAABqZozR0aNH1bJlywq3Rj8TQfcMe/bs8dzfHgAAAKFr586datWqVbXPE3TPUH4rzZ07d3ruc+9PLpdLS5cu1ZAhQxQbG+v3z4N/sI7hjzW0B9bRHlhHe/DnOhYWFiotLa3SLdDPRNA9Q3m7QmJiYsCCbkJCghITE/lhDmOsY/hjDe2BdbQH1tEeArGOZ2sz5WI0AAAA2BJBFwAAALZE0AUAAIAtEXQBAABgSwRdAAAA2BJBFwAAALZE0AUAAIAtEXQBAABgSwRdAAAA2BJ3RgMAAECtlLmNvsgr0P6jJ9WsYbx6t01SdFTNdysLJFsG3WeffVZPPPGE8vPz1b17dz3zzDPq3bt3sMsCAAAIiqoCqaQqQ2p14fXM8R+Ol+jhRZu198hJz+e0aBSvqVd10bCuLYL1rVZgu6D75ptvasKECXr++efVp08fzZ49W0OHDtWWLVvUrFmzYJcHAADgNW9DZ03jOZvzlf1exUDaOCFWknS4yOUZa9EoXiO6t9C7G/ZWCq9VjVcl/8hJ3bFgrZ674QL9rGOyxb8bvrNd0H3yySd166236je/+Y0k6fnnn9eiRYv08ssva9KkSUGuDgAARIq6htTqdkx9CaONE2IrhNlyVY3tPXJSc/6T5/V4VYwkh6Ts9zZrYPv+Xr3Gn2wVdEtKSrRmzRo98MADnrGoqChddtllWrlyZZWvKS4uVnFxsedxYWGhJMnlcsnlqvyHwGrlnxGIz4L/sI7hjzW0B9bRHkJ5HcvcRl9+/4P2Hy1Ws4ZOXdi6iSeknj5ecLxEj/57i/ILf8oYqYlOXdktVe9/le/VeFV8DaNVBVp/Mz/Ws+rbA5L8s47evqfDGGMs//Qg2bNnj8455xytWLFCmZmZnvH7779fn3zyiVavXl3pNdOmTVN2dnal8ddff10JCQl+rRcAAASO20jfFjpU6JISY6WMRKMoR9XjUuWxrwocemt7lA6X/HSxVeM4owuaurX2UMXxU3FPOrW/eeaYL+NVXdjl63hw3NS+TL2S/RMzi4qKdN111+nIkSNKTEysdp6tdnRr44EHHtCECRM8jwsLC5WWlqYhQ4bU+BtnFZfLpZycHA0ePFixsbF+/zz4B+sY/lhDe2Ad7aGqdfR2J/XC1k0kqdLYh7n79djir73aSW1cL0aSQ4dPuCqMHT5RWqnWwyUOLdsbXcV3UVXgrC6E+ns8OAZl9tKRrV/65eex/F/gz8ZWQTc5OVnR0dHat29fhfF9+/YpNTW1ytc4nU45nc5K47GxsQH9H8lAfx78g3UMf6yhPbCOocXXK/7X5hVozUGHmu46qsx2zaq8mKqmnlSp4j/ZV9enml9YrJc+/77SeJWBtooxVM0hKbVRvC7KSNEHW/3z8+jt+9kq6MbFxalXr1766KOPNGrUKEmS2+3WRx99pPHjxwe3OAAAbCIwV/xH69WtX1YbUn3pSQ1Gn2qkKt9TnnpVl5A4T9dWQVeSJkyYoDFjxujCCy9U7969NXv2bB0/ftxzCgMAAJEskCcBWHHFPyE1vKSedo5uKFxMaLug+6tf/UoHDhzQgw8+qPz8fPXo0UNLlixR8+bNg10aAAB+EcjjqqpCSA1dDp26RO3Mv3RYcY5ui0bx+sMVndWkvpM7owXS+PHjaVUAAIQ1b/tafelfrYoVZ6cicHwNo+U7rIO7pHrdJ33/sM4+jYcyWwZdAABCjT/6Wn3tX0VwWbljWpswmpnRtFJNVY1FRzl8Gg9lBF0AAM7Cl5Aq1W3X1Ze+VloDAqe6FoBAhdQz2SmM+hNBFwAQkfwVUqW67boSXq3na0itah1ragEgpIYugi4AwBbK3Eary89fzStQZrtmAW8NYNfVesEKqVLV/auSb//cj+Ai6AIAQlbtThM4df4qrQHBZ8UV/8EMqQTX8EfQBQAEnbe7rhx5FZqsvuJ/5bb9Wvrpag3p38ezMy8RUuE7gi4AwC/qcrYrpwkETiBOApB8u+K/T9skHco16hMGx1chtBF0AQBeqevxWL7sxrLr6p1wOQkACBaCLgBEsEDe9pXdWO/40tfKSQBAzQi6ABAB6IENHl93XWvT1ypxEgBQFYIuAIQpemD9L9jnr3KRFVA3BF0ACHH0wFrL360BEruuQKgg6AJAELAb63/BbA0guAKhgaALAH7Ebmzt1PY0gcT46Ernr9IaAEQugi4A+IjdWP+rbV+ry+WqdP4qrQFA5CLoAkA12I2tHSvPdpXoawVQewRdABGvzG20Oq9Aaw461DSvQJntmlUZaNmNrag2PbC+nu0KAHVB0AVgS7VrL4jWq1u/rDbQ2mk3NhB31JLYjQUQXARdAGGN9oLaCcQdtQAg2Ai6AMIGF3udQg8sAHiHoAsg5Hi7S1sVdmPpgQWAcgRdAEHjbaCtbpc2HLEbCwCBQ9AFEBB1aTuwS8iV2I0FgEAi6AKwlJ3bDtiNBYDwQtAFUCt2bjuoLtCyGwsA4YWgC+CsIq3t4GyBVmI3FgDCAUEXgIcd2g5q216QGB+tpZ+u1pD+fZTZrlmNgRYAEB4IukAEsnPbQW3bC1wulw7lGvWh3QAAbIOgC9hYXQNtqITcmnZpudgLAFAdgi5gE3boo63tRWAAAFSFoAuEGTv00VantheBAQBQFYIuEKLs0kdrZdsBAAC+IOgCIWjJxr226aOl7QAAECwEXSAEnL57u/1gkWZ/+I3MGXNCJdBWh7YDAECoIegCAVSX/tpgoO0AABDOCLqAH4Rbfy1tBwAAOyLoAhYL5f7augRadmkBAOGGoAvUUZnbaHVegdYcdOi75d/q6WXfhmx/LYEWABBJCLqAl87ejhAtbf024HXRRwsAQNUIuoAXfGlH8Bf6aAEA8A1BFzhDVbfSHff62qC3I9B2AACAbwi6wGmq2rmNcqhSyPW38t3bey9rrzbJ9Qm0AADUAkEXEcvbnVu3H1Pu2doRhnVt4b8PBwDA5gi6iEiB3rmlvxYAgMAj6CIieHOLXX/u3NJfCwBA4BF0YXtV7d76E/21AACEBoIubMXbvlsr0F8LAEBoI+jCNvzddxvlqNjecHo7wspt+7X009Ua0r+PMts1o78WAIAQQNBFWArkiQnlkfXP1/as9i5jfdom6VCuUR8uIgMAIGQQdBF2grVzSysCAADhxTZBd/v27Xr44Ye1bNky5efnq2XLlrrhhhs0efJkxcXFBbs81EGgTkzwZucWAACED9sE3a+//lput1tz5sxRu3bttHHjRt166606fvy4Zs2aFezyUEuBPDGBnVsAAOzFNkF32LBhGjZsmOdxenq6tmzZoueee46gGyaCcWJCdUeAAQCA8GeboFuVI0eOKCkpqcY5xcXFKi4u9jwuLCyUJLlcLrlcrupeZpnyzwjEZ4WyDzbt0yOLv1Z+4U9r4d++W6cmX95JQ89r7hlzl5XKXVa792cdwx9raA+soz2wjvbgz3X09j0dxhg/3g8qeLZt26ZevXpp1qxZuvXWW6udN23aNGVnZ1caf/3115WQkODPEvGjDYccevmbqB8fWb2jeuqPd1Z7txrESoUuKTFWykg0YvMWAIDwVFRUpOuuu05HjhxRYmJitfNCPuhOmjRJM2fOrHFObm6uOnXq5Hm8e/duDRgwQAMHDtRLL71U42ur2tFNS0vTwYMHa/yNs4rL5VJOTo4GDx6s2NhYv39eKChzG335/Q/af7RYyfXjdP9bGyvs5NbFmTu3LarYufWHSFxHu2EN7YF1tAfW0R78uY6FhYVKTk4+a9AN+daFiRMnKisrq8Y56enpnq/37NmjSy+9VBdffLFeeOGFs76/0+mU0+msNB4bGxvQH65Af16w+OvislA5MSFS1tHOWEN7YB3tgXW0B3+so7fvF/JBNyUlRSkpKV7N3b17ty699FL16tVL8+bNU1RU1NlfhIBZsnGv7ljgn4vLODEBAACcKeSDrrd2796tgQMHqnXr1po1a5YOHDjgeS41NTWIlUW28pMU8o+c0MOLci0JuZyYAAAAvGGboJuTk6Nt27Zp27ZtatWqVYXnQrwN2basalPgTmUAAKA2bBN0s7KyztrLi8Cxok0hVPpuAQBAeLJN0EVwnX6zh+T6Tk17d3Od2xTYuQUAAHVB0EWdWdGi4JDUPNGpP47uoYPHitm5BQAAdUbQRZ1Y2aIwbcR56tsu2YqyAAAACLqovTK3UfZ7tCgAAIDQRNCFT07vxT14tLjW7QpJ9WP1hyvPU2oiLQoAAMA/CLrwmlW9uJL06M+7sYMLAAD8iqALr1h1VzPaFAAAQKAQdHFWte3F5SQFAAAQTARdVKu8H/fzbQd8blfgJAUAABBsBF1Uqa79uLQoAACAYCPoopLa9uP+4YrOSm7opEUBAACEBIIuKqhNP65Dp3Zws/q2JdwCAICQERXsAhBavsgr8KldoTzWTr2qCyEXAACEFHZ0UeEmEFv3HfPptfTiAgCAUEXQjXC1vehs/KXt1LddMr24AAAgZBF0I1htLjor78e9d3AHAi4AAAhp9OhGqNpedCbRjwsAAMIDO7oRyteLziT6cQEAQHgh6EaQ2lx0Nv7SDLVv3pCzcQEAQNgh6EaI2l501rddijIzmvqpKgAAAP8h6EaAulx01rttkr/KAgAA8CsuRrM5LjoDAACRih1dm+OiMwAAEKkIuja3/6h3IZeLzgAAgN0QdG2uWcN4r+Zx0RkAALAbgq4NnX6MWHJ9p5Lqx6ngeEmVc7noDAAA2BVB12Z8OUaMi84AAICdEXRt5GzHiDWuF6vDJ1yex1x0BgAA7IygaxPeHCMWHxul167vo4PHirnoDAAA2B5B1ya8OUYsv7BYUQ6HRvY4J0BVAQAABA83jLAJb48R83YeAABAuCPo2oS3x4h5Ow8AACDcEXRtonfbJLVoVH2IdUhqwTFiAAAgghB0bSI6yqFb+6dX+RzHiAEAgEjExWhhrvzmEPmFJ/Xa6u8lSc6YKBWXuj1zOEYMAABEIoJuGKvq5hAOSQ9e1VnpyQ21/+hJjhEDAAARi9aFMFV+c4gzjxQzkqYs3KQjJ0o0ssc5ysxoSsgFAAARiaAbhry5OUT2e5tV5q5pBgAAgL0RdMPQ2W4OYSTtPXJSX+QVBK4oAACAEEPQDUPcHAIAAODsCLphiJtDAAAAnB1BNwxxcwgAAICzI+iGIW4OAQAAcHYE3TC1eW+hJCk+puISpjaK13M3XMDNIQAAQMTjhhFhaO+RE3pn/W5J0oJb+shVZrg5BAAAwBkIumFo3ufb5Soz6tM2SRe2oQ8XAACgKgTdMFHmNvoir0DfHzquV1dulyTdPiAjuEUBAACEMIJuGFiyca+y39tc4SYRMVEOnXSVBbEqAACA0MbFaCFuyca9umPB2kp3Qit1G9352lot2bg3SJUBAACENoJuCCtzG2W/t1mmhjnZ721WmbumGQAAAJHJlkG3uLhYPXr0kMPh0Pr164NdTq19kVdQaSf3dEbS3iMn9UVeQeCKAgAACBO2DLr333+/WrZsGewy6mz/0epDbm3mAQAARBLbBd1///vfWrp0qWbNmhXsUuqsWcPqb/Nbm3kAAACRxFanLuzbt0+33nqr3n77bSUkJHj1muLiYhUXF3seFxaeuuOYy+WSy+XyS52nK/+Mqj6rZ6uGSk10al9hcZV9ug5JqY2c6tmqYUBqRfVqWkeEB9bQHlhHe2Ad7cGf6+jtezqMMba4kskYo+HDh6tv376aMmWKtm/frrZt22rdunXq0aNHta+bNm2asrOzK42//vrrXodlf9pwyKGXvynfeD/9jmenlm1sB7e6N7XFEgIAAHilqKhI1113nY4cOaLExMRq54V80J00aZJmzpxZ45zc3FwtXbpUf//73/XJJ58oOjra66Bb1Y5uWlqaDh48WONvnFVcLpdycnI0ePBgxcbGVjnn+f98pz/mbKsw1qKRU5Mv76Sh5zX3e404O2/WEaGNNbQH1tEeWEd78Oc6FhYWKjk5+axBN+RbFyZOnKisrKwa56Snp2vZsmVauXKlnE5nhecuvPBCXX/99XrllVeqfK3T6az0GkmKjY0N6A9XTZ/nNqd2dHu3aaLrL2qtZg3j1bttkqKjHFXOR/AE+s8NrMca2gPraA+soz34Yx29fT+fg+7UqVM1duxYtW7d2ueiaiMlJUUpKSlnnff000/rkUce8Tzes2ePhg4dqjfffFN9+vTxZ4l+99m2A5KkkT3P0cge5wS5GgAAgPDg86kL77zzjjIyMvSzn/1Mr7/+eoV/9g+mc889V127dvX86tChgyQpIyNDrVq1CnJ1tXesuFTrdhyWJF3S/uyBHwAAAKf4HHTXr1+v//73vzrvvPN0zz33KDU1VXfccYf++9//+qO+iLfq20MqdRu1bpqgtKTgXxwHAAAQLmp1jm7Pnj319NNPa8+ePZo7d6527dqlvn376vzzz9ef/vQnHTlyxOo6fdamTRsZY2q8EC0cfLbtoCSpX7vkIFcCAAAQXup0wwhjjFwul0pKSmSMUZMmTfTnP/9ZaWlpevPNN62qMaJ9uvVUf27/9gRdAAAAX9Qq6K5Zs0bjx49XixYtdO+996pnz57Kzc3VJ598oq1bt2r69Om6++67ra414uw9ckLfHjiuKIeUmUHQBQAA8IXPQbdbt2666KKLlJeXp7lz52rnzp2aMWOG2rVr55lz7bXX6sCBA5YWGok+3XqqbeH8Vo3VqB7HqwAAAPjC5+PFRo8erbFjx+qcc6o/5io5OVlut7tOheGnoEvbAgAAgO983tEt78U904kTJ/TQQw9ZUhQkt9vo823lQZdjxQAAAHzlc9DNzs7WsWPHKo0XFRUpOzvbkqIgbd5bqILjJaofF62e5zYOdjkAAABhx+fWBWOMHI7Kt57dsGGDkpKSLCkqkpW5jb7IK9DfvtghSerTNkmx0XU6HAMAACAieR10mzRpIofDIYfDoQ4dOlQIu2VlZTp27Jhuv/12vxQZKZZs3Kvs9zZr75GTnrEvv/9BSzbu1bCuLYJYGQAAQPjxOujOnj1bxhiNHTtW2dnZatSokee5uLg4tWnTRpmZmX4pMhJ8sGmf7npjg8wZ44UnS3XHgrV67oYLCLsAAAA+8DrojhkzRpLUtm1bXXzxxYqN5bgrq7iN9NjiryuF3NNlv7dZg7ukKjqqctsIAAAAKvMq6BYWFioxMVHSqdv/njhxQidOnKhybvk8eO/bQofyC4urfd5I2nvkpL7IK1BmRtPAFQYAABDGvAq6TZo00d69e9WsWTM1bty4yovRyi9SKysrs7xIuyt0eTdv/9GTZ58EAAAASV4G3WXLlnlOVFi+fLlfC4pEiV52gTRrGO/fQgAAAGzEq6A7YMCAKr+GNTISjVITndpXWFxln65DUmqjePVuy/FtAAAA3vIq6P7vf//z+g3PP//8WhcTqaIc0pThnXTXGxvkkCqE3fImkalXdeFCNAAAAB94FXR79Oghh8MhY2o6F0D06NbB0POa67kbLqh0jm5qo3hNvaoLR4sBAAD4yKugm5eX5+86IGlY1xYa3CVVQ578RN8ePK77hnbQ7QPasZMLAABQC14F3datW/u7DvwoOsqh2JhTt/w9v1VjQi4AAEAteRV03333XV1++eWKjY3Vu+++W+PcESNGWFJYJCspc0uS4qKjglwJAABA+PIq6I4aNUr5+flq1qyZRo0aVe08enStUVL6Y9CNIegCAADUlldB1+12V/k1/IOgCwAAUHckqRBU3rrgJOgCAADUWq2S1EcffaQrr7xSGRkZysjI0JVXXqkPP/zQ6toilmdHNzo6yJUAAACEL5+D7l/+8hcNGzZMDRs21D333KN77rlHiYmJGj58uJ599ll/1BhxaF0AAACoO696dE/36KOP6qmnntL48eM9Y3fffbf69u2rRx99VOPGjbO0wEjjdhuVuk/dmIOgCwAAUHs+J6nDhw9r2LBhlcaHDBmiI0eOWFJUJCvvz5Wk2GjO0AUAAKgtn4PuiBEjtHDhwkrj77zzjq688kpLiopkxaU/BV12dAEAAGrPq9aFp59+2vN1ly5dNH36dH388cfKzMyUJK1atUqff/65Jk6c6J8qI0jJ6UGXG0YAAADUmldB96mnnqrwuEmTJtq8ebM2b97sGWvcuLFefvllTZkyxdoKI4zrtLuiORy0LgAAANSWV0E3Ly/P33XgR5y4AAAAYA3SVIgpvxiNoAsAAFA3Ph8vJkm7du3Su+++qx07dqikpKTCc08++aQlhUWqn24WQdAFAACoC5+D7kcffaQRI0YoPT1dX3/9tbp27art27fLGKMLLrjAHzVGlGJaFwAAACzhc5p64IEH9Lvf/U5fffWV4uPj9a9//Us7d+7UgAEDdM011/ijxohCjy4AAIA1fE5Tubm5uummmyRJMTExOnHihBo0aKCHHnpIM2fOtLzASFNSRusCAACAFXxOU/Xr1/f05bZo0ULffvut57mDBw9aV1mEYkcXAADAGj736F500UX67LPP1LlzZw0fPlwTJ07UV199pbfeeksXXXSRP2qMKARdAAAAa/gcdJ988kkdO3ZMkpSdna1jx47pzTffVPv27TlxwQIlZWWSJCdBFwAAoE58Drrp6emer+vXr6/nn3/e0oIiHceLAQAAWKNW5+hK0pdffqnc3FxJUpcuXdSrVy/LiopktC4AAABYw+egu2vXLl177bX6/PPP1bhxY0nS4cOHdfHFF+uNN95Qq1atrK4xopSfoxvLji4AAECd+JymbrnlFrlcLuXm5qqgoEAFBQXKzc2V2+3WLbfc4o8aIwq3AAYAALCGzzu6n3zyiVasWKGOHTt6xjp27KhnnnlG/fv3t7S4SETrAgAAgDV8TlNpaWlyuVyVxsvKytSyZUtLiopkLm4YAQAAYAmf09QTTzyhu+66S19++aVn7Msvv9Q999yjWbNmWVpcJCrf0eV4MQAAgLrxqnWhSZMmcjgcnsfHjx9Xnz59FBNz6uWlpaWKiYnR2LFjNWrUKL8UGiloXQAAALCGV0F39uzZfi4D5UpoXQAAALCEV0F3zJgx/q4DPypmRxcAAMAStbphRFlZmd5++23PDSPOO+88jRgxQtHR0ZYWF4loXQAAALCGz0F327ZtGj58uHbv3u05Yuyxxx5TWlqaFi1apIyMDMuLjCQEXQAAAGv4nKbuvvtuZWRkaOfOnVq7dq3Wrl2rHTt2qG3btrr77rv9UaNPFi1apD59+qhevXpq0qRJ2F0cR48uAACANWp1w4hVq1YpKSnJM9a0aVPNmDFDffv2tbQ4X/3rX//SrbfeqkcffVSDBg1SaWmpNm7cGNSafMWOLgAAgDV8DrpOp1NHjx6tNH7s2DHFxcVZUlRtlJaW6p577tETTzyhm2++2TPepUuXGl9XXFys4uJiz+PCwkJJksvlqvLGGFYr/4zy/xa7yiRJ0TIB+XxY48x1RPhhDe2BdbQH1tEe/LmO3r6nz0H3yiuv1G233aa5c+eqd+/ekqTVq1fr9ttv14gRI3x9O8usXbtWu3fvVlRUlHr27Kn8/Hz16NFDTzzxhLp27Vrt6x577DFlZ2dXGl+6dKkSEhL8WXIFOTk5kqT9h6IlOfS/9WtV9r0J2OfDGuXriPDFGtoD62gPrKM9+GMdi4qKvJrnMMb4lKYOHz6sMWPG6L333lNsbKykU7upI0aM0Pz589WoUSPfq7XAG2+8oWuvvVbnnnuunnzySbVp00Z//OMftXTpUn3zzTcVWi1OV9WOblpamg4ePKjExES/1+1yuZSTk6PBgwcrNjZWV/55hbbsO6b5Wb3UN6Op3z8f1jhzHRF+WEN7YB3tgXW0B3+uY2FhoZKTk3XkyJEa85pPO7rGGBUWFuqNN97Q7t27PceLde7cWe3atatbxdWYNGmSZs6cWeOc3Nxcud2nelsnT56sX/ziF5KkefPmqVWrVvrHP/6h3/72t1W+1ul0yul0VhqPjY0N6A9X+ee5yk79vaNeXGA/H9YI9J8bWI81tAfW0R5YR3vwxzp6+34+B9127dpp06ZNat++vd/C7ekmTpyorKysGuekp6dr7969kir25DqdTqWnp2vHjh3+LNFS3DACAADAGj4F3aioKLVv316HDh1S+/bt/VVTBSkpKUpJSTnrvF69esnpdGrLli3q16+fpFNb5tu3b1fr1q39XaZlPMeLEXQBAADqxOc0NWPGDN13330hd2xXYmKibr/9dk2dOlVLly7Vli1bdMcdd0iSrrnmmiBX573y48WcBF0AAIA68fnUhZtuuklFRUXq3r274uLiVK9evQrPFxQUWFacr5544gnFxMToxhtv1IkTJ9SnTx8tW7ZMTZo0CVpNvnJ5bhjB7ZQBAADqwueg+9RTT8nhcPijljqLjY3VrFmzNGvWrGCXUmvcMAIAAMAaPgfda6+9VqWlpapfv74/6olobrdRqfvUqQsEXQAAgLrxOk0dOHBAl19+uRo0aKDExERddNFF2rZtmz9rizjlF6JJBF0AAIC68jpN/f73v9f69ev10EMPadasWTp8+LBuvfVWf9YWccqPFpOkuGiCLgAAQF143bqQk5Oj+fPna+jQoZJO3Qq4c+fOKi4urvKGC/BdyWlBNzY6NPugAQAAwoXX24Z79uxR9+7dPY/bt28vp9PpuVED6u70M3RD9YI/AACAcOHTv49Hn3HkVXR0tIwxlhYUyTxn6NK2AAAAUGdety4YY9ShQ4cKO43Hjh1Tz549FRX1UzAL5jm64Y6jxQAAAKzjddCdN2+eP+uACLoAAABW8jrojhkzxp91QFJJWZkkgi4AAIAVSFQhpPx4sVh6dAEAAOqMRBVCPK0LBF0AAIA6I1GFEHp0AQAArEOiCiGnn6MLAACAuiFRhRDXj0HXSdAFAACoM69PXSg3YcKEKscdDofi4+PVrl07jRw5UklJSXUuLtLQowsAAGAdn4PuunXrtHbtWpWVlaljx46SpG+++UbR0dHq1KmT/vKXv2jixIn67LPP1KVLF8sLtjN6dAEAAKzjc6IaOXKkLrvsMu3Zs0dr1qzRmjVrtGvXLg0ePFjXXnutdu/erUsuuUT33nuvP+q1tWKCLgAAgGV8TlRPPPGEHn74YSUmJnrGGjVqpGnTpunxxx9XQkKCHnzwQa1Zs8bSQiOB52I0WhcAAADqzOdEdeTIEe3fv7/S+IEDB1RYWChJaty4sUpKSupeXYShdQEAAMA6tWpdGDt2rBYuXKhdu3Zp165dWrhwoW6++WaNGjVKkvTFF1+oQ4cOVtdqewRdAAAA6/h8MdqcOXN077336te//rVKS0tPvUlMjMaMGaOnnnpKktSpUye99NJL1lYaAQi6AAAA1vE56DZo0EAvvviinnrqKX333XeSpPT0dDVo0MAzp0ePHpYVGEnKe3Sd9OgCAADUmc9Bt1yDBg08Z+WeHnJRe+zoAgAAWMfnROV2u/XQQw+pUaNGat26tVq3bq3GjRvr4Ycfltvt9keNEYOgCwAAYB2fd3QnT56suXPnasaMGerbt68k6bPPPtO0adN08uRJTZ8+3fIiI0Uxx4sBAABYxueg+8orr+ill17SiBEjPGPnn3++zjnnHN15550E3Too39GNZUcXAACgznxOVAUFBerUqVOl8U6dOqmgoMCSoiKVp3WBHV0AAIA68zlRde/eXX/+858rjf/5z39W9+7dLSkqUtGjCwAAYB2fWxcef/xxXXHFFfrwww+VmZkpSVq5cqV27typxYsXW15gJHGVHy9G0AUAAKgznxPVgAED9M033+jnP/+5Dh8+rMOHD+vqq6/Wli1b1L9/f3/UGDHKz9FlRxcAAKDuanWObsuWLStddLZr1y7ddttteuGFFywpLBL91KMbHeRKAAAAwp9lW4eHDh3S3LlzrXq7iESPLgAAgHVIVCGkmKALAABgGRJVCCnhhhEAAACWIVGFEFoXAAAArOP1xWhXX311jc8fPny4rrVEvPKgy/FiAAAAded10G3UqNFZn7/pppvqXFAk43gxAAAA63gddOfNm+fPOiJemduozG0k0aMLAABgBRJViChvW5DY0QUAALACiSpElLctSARdAAAAK5CoQsTpO7oxUY4gVgIAAGAPBN0QcfqFaA4HQRcAAKCuCLohwnO0GBeiAQAAWIJUFSK4WQQAAIC1SFUhwlX249FiBF0AAABLkKpCBDeLAAAAsBapKkR4Whfo0QUAALAEqSpEsKMLAABgLVJViOBiNAAAAGvZKlV98803GjlypJKTk5WYmKh+/fpp+fLlwS7LK7QuAAAAWMtWqerKK69UaWmpli1bpjVr1qh79+668sorlZ+fH+zSzorWBQAAAGvZJlUdPHhQW7du1aRJk3T++eerffv2mjFjhoqKirRx48Zgl3dWnhtGEHQBAAAsERPsAqzStGlTdezYUa+++qouuOACOZ1OzZkzR82aNVOvXr2qfV1xcbGKi4s9jwsLCyVJLpdLLpfL73WXf8aJklP/jYlyBORzYa3yNWPtwhdraA+soz2wjvbgz3X09j0dxhhj+acHya5duzRq1CitXbtWUVFRatasmRYtWqSePXtW+5pp06YpOzu70vjrr7+uhIQEf5ZbwfI9Dr39fbR6Jbt1U3t3wD4XAAAg3BQVFem6667TkSNHlJiYWO28kA+6kyZN0syZM2uck5ubq44dO2rUqFFyuVyaPHmy6tWrp5deeknvvvuu/vvf/6pFixZVvraqHd20tDQdPHiwxt84q7hcLuXk5CivXgfNXvadfnnBOXrs5+f5/XNhrfJ1HDx4sGJjY4NdDmqBNbQH1tEeWEd78Oc6FhYWKjk5+axBN+RbFyZOnKisrKwa56Snp2vZsmV6//339cMPP3i+4b/85S/KycnRK6+8okmTJlX5WqfTKafTWWk8NjY2oD9cP94BWM7YaH6ow1ig/9zAeqyhPbCO9sA62oM/1tHb9wv5oJuSkqKUlJSzzisqKpIkRUVVvJgrKipKbnfotwKUlJ5Kupy6AAAAYA3bpKrMzEw1adJEY8aM0YYNG/TNN9/ovvvuU15enq644opgl3dWHC8GAABgLdukquTkZC1ZskTHjh3ToEGDdOGFF+qzzz7TO++8o+7duwe7vLPyHC/GDSMAAAAsEfKtC7648MIL9cEHHwS7jFpxsaMLAABgKVJViPDcApigCwAAYAlSVYjw9OjSugAAAGAJUlWI+GlHNzrIlQAAANgDQTdEcOoCAACAtUhVIYIeXQAAAGuRqkJEyY+3RqNHFwAAwBqkqhDhOUeXHV0AAABLkKpCBK0LAAAA1iJVhQguRgMAALAWqSpEeHZ06dEFAACwBKkqRJTv6MYSdAEAACxBqgoR9OgCAABYi1QVIsp3dDl1AQAAwBqkqhDBji4AAIC1SFUhwG1O/ZK4GA0AAMAqpKoQ8ONmriR2dAEAAKxCqgoBpeanrwm6AAAA1iBVhYDyHV2HQ4qJcgS3GAAAAJsg6IaA0tP6cx0Ogi4AAIAVCLohoHxHl7YFAAAA65CsQkB50OUMXQAAAOuQrEJAKUeLAQAAWI5kFQJoXQAAALAeySoElJpTF6ARdAEAAKxDsgoB7OgCAABYj2QVAsp7dGPp0QUAALAMySoEeHZ0CboAAACWIVmFAM+pC7QuAAAAWIZkFQI4RxcAAMB6JKsQwMVoAAAA1iNZhYAybhgBAABgOZJVCKBHFwAAwHokqxBQ6uaGEQAAAFYjWYWAn44Xiw5uIQAAADZC0A0BtC4AAABYj2QVAjh1AQAAwHokqxBQvqPLOboAAADWIVmFAG4BDAAAYD2SVQigRxcAAMB6JKsQQI8uAACA9UhWIaA86MbSugAAAGAZklUIoHUBAADAeiSrEOC5Mxo7ugAAAJYhWYUAjhcDAACwHskqBHAxGgAAgPVIViGgjB5dAAAAy5GsQgA3jAAAALAeySoEcOoCAACA9UhWIYAeXQAAAOuRrEKAZ0eX1gUAAADLhE2ymj59ui6++GIlJCSocePGVc7ZsWOHrrjiCiUkJKhZs2a67777VFpaGthCa6F8R5fjxQAAAKwTE+wCvFVSUqJrrrlGmZmZmjt3bqXny8rKdMUVVyg1NVUrVqzQ3r17ddNNNyk2NlaPPvpoECr2TmmZW0Y/3jCCoAsAAGCZsElW2dnZuvfee9WtW7cqn1+6dKk2b96sBQsWqEePHrr88sv18MMP69lnn1VJSUmAq/VeSZnb8zVBFwAAwDphs6N7NitXrlS3bt3UvHlzz9jQoUN1xx13aNOmTerZs2eVrysuLlZxcbHncWFhoSTJ5XLJ5XL5t2hJx0/+FMId7jK5XMbvnwnrlf9ZCcSfGfgHa2gPrKM9sI724M919PY9bRN08/PzK4RcSZ7H+fn51b7uscceU3Z2dqXxpUuXKiEhwdoiq3CkRJJi5JDR0g+W+P3z4F85OTnBLgF1xBraA+toD6yjPfhjHYuKiryaF9SgO2nSJM2cObPGObm5uerUqZPfanjggQc0YcIEz+PCwkKlpaVpyJAhSkxM9Nvnltt+oFBas0rOmGgNHz7U758H/3C5XMrJydHgwYMVGxsb7HJQC6yhPbCO9sA62oM/17H8X+DPJqhBd+LEicrKyqpxTnp6ulfvlZqaqi+++KLC2L59+zzPVcfpdMrpdFYaj42NDcgPl1vRpz4vJoofZhsI1J8b+A9raA+soz2wjvbgj3X09v2CGnRTUlKUkpJiyXtlZmZq+vTp2r9/v5o1aybp1FZ5YmKiunTpYsln+EP5xWicoQsAAGCtsOnR3bFjhwoKCrRjxw6VlZVp/fr1kqR27dqpQYMGGjJkiLp06aIbb7xRjz/+uPLz8zVlyhSNGzeuyh3bUFHy4yG6nLgAAABgrbAJug8++KBeeeUVz+PyUxSWL1+ugQMHKjo6Wu+//77uuOMOZWZmqn79+hozZoweeuihYJXsFXZ0AQAA/CNsgu78+fM1f/78Gue0bt1aixcvDkxBFnGVB90YR5ArAQAAsBe2EYOM1gUAAAD/IF0FWUnpqRtE0LoAAABgLdJVkHl6dNnRBQAAsBTpKsg8rQvs6AIAAFiKdBVk7OgCAAD4B+kqyNjRBQAA8A/SVZCxowsAAOAfpKsg43gxAAAA/yBdBRmtCwAAAP5BugoyWhcAAAD8g3QVZOU7urHR3AIYAADASgTdIPPs6NK6AAAAYCnSVZB5bgFM6wIAAIClSFdBxqkLAAAA/kG6CjJaFwAAAPyDdBVkLk5dAAAA8AvSVZBxji4AAIB/kK6CjHN0AQAA/IN0FWTs6AIAAPgH6SrIftrR5YYRAAAAViLoBhnHiwEAAPgH6SrIPDeMoHUBAADAUqSrIONiNAAAAP8gXQUZF6MBAAD4B+kqyNjRBQAA8A/SVZCV7+jGsqMLAABgKdJVEBlj2NEFAADwE9JVEJW6jcypQxfo0QUAALAY6SqIytsWJG4YAQAAYDWCbhBVCLrs6AIAAFiKdBVErh/7cx0yiiHoAgAAWIp0FUTFP+7och0aAACA9YhYQVR+4gLtuQAAANYj6AZRCTu6AAAAfkPECqITJWWSJLdbWp1XoDK3CXJFAAAA9kHQDZIlG/fq1le/lCQdL3Pohpe/VL+Zy7Rk494gVwYAAGAPBN0gWLJxr+5YsFaHjpdUGM8/clJ3LFhL2AUAALAAQTfAytxG2e9tVlVNCuVj2e9tpo0BAACgjgi6AfZFXoH2HjlZ7fNG0t4jJ/VFXkHgigIAALAhgm6A7T9afcitzTwAAABUjaAbYM0axls6DwAAAFUj6AZY77ZJatEoXtXdI8IhqUWjePVumxTIsgAAAGyHoBtg0VEOTb2qiyRVCrvlj6de1UXRUdwuDQAAoC4IukEwrGsLPXfDBUptVLE9IbVRvJ674QIN69oiSJUBAADYR0ywC4hUw7q20OAuqVq5bb+WfrpaQ/r3UWa7ZuzkAgAAWISgG0TRUQ71aZukQ7lGfdomEXIBAAAsROsCAAAAbImgCwAAAFsi6AIAAMCWCLoAAACwpbAJutOnT9fFF1+shIQENW7cuNLzGzZs0LXXXqu0tDTVq1dPnTt31p/+9KfAFwoAAICQEDanLpSUlOiaa65RZmam5s6dW+n5NWvWqFmzZlqwYIHS0tK0YsUK3XbbbYqOjtb48eODUDEAAACCKWyCbnZ2tiRp/vz5VT4/duzYCo/T09O1cuVKvfXWWwRdAACACBQ2Qbc2jhw5oqSkpBrnFBcXq7i42PO4sLBQkuRyueRyufxaX/nnnP5fhCfWMfyxhvbAOtoD62gP/lxHb9/TtkF3xYoVevPNN7Vo0aIa5z322GOe3eLTLV26VAkJCf4qr5KcnJyAfRb8h3UMf6yhPbCO9sA62oM/1rGoqMireUENupMmTdLMmTNrnJObm6tOnTr59L4bN27UyJEjNXXqVA0ZMqTGuQ888IAmTJjgeVxYWKi0tDQNGTJEiYmJPn1ubbhcLuXk5Gjw4MGKjY31++fBP1jH8Mca2gPraA+soz34cx3L/wX+bIIadCdOnKisrKwa56Snp/v0nps3b9bPfvYz3XbbbZoyZcpZ5zudTjmdTs9jY4wk6cSJEwH54XK5XCoqKtKJEydUWlrq98+Df7CO4Y81tAfW0R5YR3vw5zqeOHFC0k+5rTpBDbopKSlKSUmx7P02bdqkQYMGacyYMZo+fXqt3uPo0aOSpLS0NMvqAgAAgPWOHj2qRo0aVft82PTo7tixQwUFBdqxY4fKysq0fv16SVK7du3UoEEDbdy4UYMGDdLQoUM1YcIE5efnS5Kio6N9CtMtW7bUzp071bBhQzkcDn98KxWUt0rs3LkzIK0S8A/WMfyxhvbAOtoD62gP/lxHY4yOHj2qli1b1jgvbILugw8+qFdeecXzuGfPnpKk5cuXa+DAgfrnP/+pAwcOaMGCBVqwYIFnXuvWrbV9+3avPycqKkqtWrWyrG5vJSYm8sNsA6xj+GMN7YF1tAfW0R78tY417eSWC5s7o82fP1/GmEq/Bg4cKEmaNm1alc/7EnIBAABgH2ETdAEAAABfEHSDzOl0aurUqRVOfkD4YR3DH2toD6yjPbCO9hAK6+gwZzuXAQAAAAhD7OgCAADAlgi6AAAAsCWCLgAAAGyJoAsAAABbIugGwLPPPqs2bdooPj5effr00RdffFHj/H/84x/q1KmT4uPj1a1bNy1evDhAlaImvqzjiy++qP79+6tJkyZq0qSJLrvssrOuO/zP15/Fcm+88YYcDodGjRrl3wLhFV/X8fDhwxo3bpxatGghp9OpDh068L+rIcDXdZw9e7Y6duyoevXqKS0tTffee69OnjwZoGpRlf/85z+66qqr1LJlSzkcDr399ttnfc3HH3+sCy64QE6nU+3atdP8+fP9W6SBX73xxhsmLi7OvPzyy2bTpk3m1ltvNY0bNzb79u2rcv7nn39uoqOjzeOPP242b95spkyZYmJjY81XX30V4MpxOl/X8brrrjPPPvusWbduncnNzTVZWVmmUaNGZteuXQGuHOV8XcNyeXl55pxzzjH9+/c3I0eODEyxqJav61hcXGwuvPBCM3z4cPPZZ5+ZvLw88/HHH5v169cHuHKcztd1fO2114zT6TSvvfaaycvLMx988IFp0aKFuffeewNcOU63ePFiM3nyZPPWW28ZSWbhwoU1zv/uu+9MQkKCmTBhgtm8ebN55plnTHR0tFmyZInfaiTo+lnv3r3NuHHjPI/LyspMy5YtzWOPPVbl/NGjR5srrriiwlifPn3Mb3/7W7/WiZr5uo5nKi0tNQ0bNjSvvPKKv0rEWdRmDUtLS83FF19sXnrpJTNmzBiCbgjwdR2fe+45k56ebkpKSgJVIrzg6zqOGzfODBo0qMLYhAkTTN++ff1aJ7znTdC9//77zXnnnVdh7Fe/+pUZOnSo3+qidcGPSkpKtGbNGl122WWesaioKF122WVauXJlla9ZuXJlhfmSNHTo0Grnw/9qs45nKioqksvlUlJSkr/KRA1qu4YPPfSQmjVrpptvvjkQZeIsarOO7777rjIzMzVu3Dg1b95cXbt21aOPPqqysrJAlY0z1GYdL774Yq1Zs8bT3vDdd99p8eLFGj58eEBqhjWCkXFi/PbO0MGDB1VWVqbmzZtXGG/evLm+/vrrKl+Tn59f5fz8/Hy/1Yma1WYdz/T73/9eLVu2rPQDjsCozRp+9tlnmjt3rtavXx+ACuGN2qzjd999p2XLlun666/X4sWLtW3bNt15551yuVyaOnVqIMrGGWqzjtddd50OHjyofv36yRij0tJS3X777fq///u/QJQMi1SXcQoLC3XixAnVq1fP8s9kRxfwsxkzZuiNN97QwoULFR8fH+xy4IWjR4/qxhtv1Isvvqjk5ORgl4M6cLvdatasmV544QX16tVLv/rVrzR58mQ9//zzwS4NPvj444/16KOP6i9/+YvWrl2rt956S4sWLdLDDz8c7NIQ4tjR9aPk5GRFR0dr3759Fcb37dun1NTUKl+Tmprq03z4X23WsdysWbM0Y8YMffjhhzr//PP9WSZq4Osafvvtt9q+fbuuuuoqz5jb7ZYkxcTEaMuWLcrIyPBv0aikNj+LLVq0UGxsrKKjoz1jnTt3Vn5+vkpKShQXF+fXmlFZbdbxD3/4g2688UbdcsstkqRu3brp+PHjuu222zR58mRFRbFvFw6qyziJiYl+2c2V2NH1q7i4OPXq1UsfffSRZ8ztduujjz5SZmZmla/JzMysMF+ScnJyqp0P/6vNOkrS448/rocfflhLlizRhRdeGIhSUQ1f17BTp0766quvtH79es+vESNG6NJLL9X69euVlpYWyPLxo9r8LPbt21fbtm3z/EVFkr755hu1aNGCkBsktVnHoqKiSmG2/C8vp66DQjgISsbx22VuMMacOkLF6XSa+fPnm82bN5vbbrvNNG7c2OTn5xtjjLnxxhvNpEmTPPM///xzExMTY2bNmmVyc3PN1KlTOV4sBPi6jjNmzDBxcXHmn//8p9m7d6/n19GjR4P1LUQ8X9fwTJy6EBp8XccdO3aYhg0bmvHjx5stW7aY999/3zRr1sw88sgjwfoWYHxfx6lTp5qGDRuav/3tb+a7774zS5cuNRkZGWb06NHB+hZgjDl69KhZt26dWbdunZFknnzySbNu3Trz/fffG2OMmTRpkrnxxhs988uPF7vvvvtMbm6uefbZZzlezA6eeeYZc+6555q4uDjTu3dvs2rVKs9zAwYMMGPGjKkw/+9//7vp0KGDiYuLM+edd55ZtGhRgCtGVXxZx9atWxtJlX5NnTo18IXDw9efxdMRdEOHr+u4YsUK06dPH+N0Ok16erqZPn26KS0tDXDVOJMv6+hyucy0adNMRkaGiY+PN2lpaebOO+80P/zwQ+ALh8fy5cur/P+68rUbM2aMGTBgQKXX9OjRw8TFxZn09HQzb948v9boMIY9fwAAANgPPboAAACwJYIuAAAAbImgCwAAAFsi6AIAAMCWCLoAAACwJYIuAAAAbImgCwAAAFsi6AIAAMCWCLoAYHMOh0Nvv/12sMsAgIAj6AJACMvKytKoUaOCXQYAhCWCLgAAAGyJoAsAYWLgwIG6++67df/99yspKUmpqamaNm1ahTlbt27VJZdcovj4eHXp0kU5OTmV3mfnzp0aPXq0GjdurKSkJI0cOVLbt2+XJH399ddKSEjQ66+/7pn/97//XfXq1dPmzZv9+e0BgOUIugAQRl555RXVr19fq1ev1uOPP66HHnrIE2bdbreuvvpqxcXFafXq1Xr++ef1+9//vsLrXS6Xhg4dqoYNG+rTTz/V559/rgYNGmjYsGEqKSlRp06dNGvWLN15553asWOHdu3apdtvv10zZ85Uly5dgvEtA0CtOYwxJthFAACqlpWVpcOHD+vtt9/WwIEDVVZWpk8//dTzfO/evTVo0CDNmDFDS5cu1RVXXKHvv/9eLVu2lCQtWbJEl19+uRYuXKhRo0ZpwYIFeuSRR5SbmyuHwyFJKikpUePGjfX2229ryJAhkqQrr7xShYWFiouLU3R0tJYsWeKZDwDhIibYBQAAvHf++edXeNyiRQvt379fkpSbm6u0tDRPyJWkzMzMCvM3bNigbdu2qWHDhhXGT548qW+//dbz+OWXX1aHDh0UFRWlTZs2EXIBhCWCLgCEkdjY2AqPHQ6H3G63168/duyYevXqpddee63ScykpKZ6vN2zYoOPHjysqKkp79+5VixYtal80AAQJQRcAbKJz587auXNnhWC6atWqCnMuuOACvfnmm2rWrJkSExOrfJ+CggJlZWVp8uTJ2rt3r66//nqtXbtW9erV8/v3AABW4mI0ALCJyy67TB06dNCYMWO0YcMGffrpp5o8eXKFOddff72Sk5M1cuRIffrpp8rLy9PHH3+su+++W7t27ZIk3X777UpLS9OUKVP05JNPqqysTL/73e+C8S0BQJ0QdAHAJqKiorRw4UKdOHFCvXv31i233KLp06dXmJOQkKD//Oc/Ovfcc3X11Verc+fOuvnmm3Xy5EklJibq1Vdf1eLFi/XXv/5VMTExql+/vhYsWKAXX3xR//73v4P0nQFA7XDqAgAAAGyJHV0AAADYEkEXAAAAtkTQBQAAgC0RdAEAAGBLBF0AAADYEkEXAAAAtkTQBQAAgC0RdAEAAGBLBF0AAADYEkEXAAAAtkTQBQAAgC39f0Sziu1h24teAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "probs = np.arange(0.00001, 1, 0.01)\n",
    "log_ps = np.log(probs)\n",
    "\n",
    "plt.figure(figsize=(8, 5))\n",
    "plt.plot(probs, log_ps, marker='o')\n",
    "plt.title('Log Probabilities')\n",
    "plt.xlabel('Index')\n",
    "plt.ylabel('Log Probability')\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "f36b37f9-e9cf-48d6-a29a-084fd4275b9e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:45:38.433110Z",
     "iopub.status.busy": "2024-12-06T12:45:38.432761Z",
     "iopub.status.idle": "2024-12-06T12:45:38.624547Z",
     "shell.execute_reply": "2024-12-06T12:45:38.622897Z",
     "shell.execute_reply.started": "2024-12-06T12:45:38.433086Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArMAAAHWCAYAAABkNgFvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABe/ElEQVR4nO3deViU5f7H8c+wDSCCCwouJO655ZpblloqaUezRU0tl8o2UU/8OqaVopXHVrPMtCytc8qjtlh2KndtMc2TW7lrapoKoqgoICDz/P4wJka2GWCYGeb9ui4vmWfueeaGe9SPz3zne5sMwzAEAAAAeCAfV08AAAAAKC7CLAAAADwWYRYAAAAeizALAAAAj0WYBQAAgMcizAIAAMBjEWYBAADgsQizAAAA8FiEWQAAAHgswiwAj3HkyBGZTCa98sorpXbOKVOmyGQy6fTp00WOjY6O1ogRI6y3169fL5PJpPXr11uPjRgxQtHR0Q49Nxz3/vvvy2Qy6eeffy61c3br1k3NmzcvclzO6/D999+3HstvLa9+vRT13N26dXNgtgByEGYBD+WMf8yLK+cf8pxfwcHBatq0qZ555hmlpKS4enoulZaWpilTptgE3vKsW7duNq+FKlWq6Prrr9f8+fNlsVhcPT2X2r17t6ZMmaIjR464eipAueLn6gkAKD/mzJmjkJAQXbx4UStXrtS0adO0du1abdiwoVxcgdy3b598fAq/BjBv3jyb0JaWlqapU6dKUp4rb88884wmTJhQ6vN0tdq1a2v69OmSpKSkJP3rX//SAw88oP379+uFF15w8exKrk6dOkpPT5e/v3+h465+vezevVtTp05Vt27d8ly9X7lypTOmCngFwiyAUnP33XcrPDxckvTII4/orrvu0meffaZNmzapU6dO+T4mLS1NwcHBZTnNYjObzUWOKSrg5Obn5yc/v/L313BYWJjuvfde6+2HH35YjRs31ptvvqnnnnsu35+RxWJRZmamAgMDy3KqxWIymeyapz2vlxwBAQElmRLg1SgzAMq5bdu2qXfv3goNDVVISIhuueUWbdq0Kc+4X375RV27dlVQUJBq166t559/XgsWLJDJZCr226I333yzJOnw4cOS/qpJ3LJli2666SYFBwfrqaeekiSdOnVKDzzwgCIiIhQYGKiWLVvqgw8+KPDcr732murUqaOgoCB17dpVO3fuzPP9jBgxQvXq1VNgYKAiIyN1//3368yZM/me7/Tp0xo4cKBCQ0NVtWpVjRs3TpcuXbIZY08NZO6a2SNHjqhatWqSpKlTp1rfep8yZYqkgmtmP/zwQ7Vt21ZBQUGqUqWK7rnnHh07dsxmzIEDB3TXXXcpMjJSgYGBql27tu655x6dP3++wLnFxsYqJCREaWlpee4bPHiwIiMjlZ2dLUn6+eefFRMTo/DwcAUFBalu3bq6//77C/3eCxIcHKyOHTsqNTVVSUlJkq4EwtjYWH300Udq1qyZzGazli9fLsn+16x05T9DDz/8sKpWrarQ0FANGzZMZ8+etRnzxRdf6LbbblPNmjVlNptVv359Pffcc9bv9WpbtmxR586drd/33Llzbe7Pr2Y2P7lfL++//74GDBggSerevbv1tZBTfpJfzWxGRobi4+PVoEEDmc1mRUVFafz48crIyLAZt2rVKnXp0kWVKlVSSEiIGjdubP1zBXiD8ndJAIDVrl27dOONNyo0NFTjx4+Xv7+/3n77bXXr1k3ffvutOnToIEk6fvy49R/YiRMnqkKFCnr33XcdurKUn99++02SVLVqVeuxM2fOqHfv3rrnnnt07733KiIiQunp6erWrZsOHjyo2NhY1a1bVx9//LFGjBihc+fOady4cTbn/de//qULFy5o9OjRunTpkl5//XXdfPPN+vXXXxURESHpyj/whw4d0siRIxUZGaldu3bpnXfe0a5du7Rp06Y8IXLgwIGKjo7W9OnTtWnTJr3xxhs6e/as/vWvfxX7+69WrZrmzJmjRx99VHfccYfuvPNOSdJ1111X4GOmTZumSZMmaeDAgXrwwQeVlJSkWbNm6aabbtK2bdtUqVIlZWZmKiYmRhkZGRozZowiIyN1/Phx/fe//9W5c+cUFhaW77kHDRqk2bNn66uvvrIGK+lKIPzyyy81YsQI+fr66tSpU+rVq5eqVaumCRMmqFKlSjpy5Ig+++yzYv8sDh06JF9fX1WqVMl6bO3atVqyZIliY2MVHh6u6Ohou1+zOWJjY1WpUiVNmTJF+/bt05w5c/T7779bP5wnXQmSISEhiouLU0hIiNauXavJkycrJSVFL7/8ss35zp49qz59+mjgwIEaPHiwlixZokcffVQBAQHFDvOSdNNNN2ns2LF644039NRTT6lJkyaSZP39ahaLRf369dMPP/yghx56SE2aNNGvv/6q1157Tfv379fnn38u6cqf8b/97W+67rrr9Oyzz8psNuvgwYPasGFDsecKeBwDgEdasGCBIcn43//+V+CY/v37GwEBAcZvv/1mPXbixAmjYsWKxk033WQ9NmbMGMNkMhnbtm2zHjtz5oxRpUoVQ5Jx+PDhQucSHx9vSDL27dtnJCUlGYcPHzbefvttw2w2GxEREUZqaqphGIbRtWtXQ5Ixd+5cm8fPnDnTkGR8+OGH1mOZmZlGp06djJCQECMlJcUwDMM4fPiwIckICgoy/vjjD+vYn376yZBkPP7449ZjaWlpeeb5n//8x5BkfPfdd3nm3q9fP5uxjz32mCHJ2LFjh/VYnTp1jOHDh1tvr1u3zpBkrFu3znps+PDhRp06day3k5KSDElGfHx8gT+3HEeOHDF8fX2NadOm2Yz79ddfDT8/P+vxbdu2GZKMjz/+OM85C2OxWIxatWoZd911l83xJUuW2Pxcli5dWuRrqyBdu3Y1rr32WiMpKclISkoy9uzZY4wdO9aQZPTt29c6TpLh4+Nj7Nq1y+bx9r5mc17/bdu2NTIzM63HX3rpJUOS8cUXX1iP5fdaePjhh43g4GDj0qVLNnOXZLz66qvWYxkZGUarVq2M6tWrW58n53W4YMEC67ir19Iw8r5ePv744zyvl9zP3bVrV+vtf//734aPj4/x/fff24ybO3euIcnYsGGDYRiG8dprrxmSjKSkpDznBLwFZQZAOZWdna2VK1eqf//+qlevnvV4jRo1NGTIEP3www/WTgPLly9Xp06d1KpVK+u4KlWqaOjQoQ49Z+PGjVWtWjXVrVtXDz/8sBo0aKCvvvrKpibWbDZr5MiRNo/7+uuvFRkZqcGDB1uP+fv7a+zYsbp48aK+/fZbm/H9+/dXrVq1rLfbt2+vDh066Ouvv7YeCwoKsn596dIlnT59Wh07dpQkbd26Nc/cR48ebXN7zJgx1rmVlc8++0wWi0UDBw7U6dOnrb8iIyPVsGFDrVu3TpKsV15XrFiRb8lAQUwmkwYMGKCvv/5aFy9etB5fvHixatWqpS5dukiS9erpf//7X2VlZTn8fezdu1fVqlVTtWrV1KRJE82aNUu33Xab5s+fbzOua9euatq0qfW2I6/ZHA899JBNDe6jjz4qPz+/Al8LFy5c0OnTp3XjjTcqLS1Ne/futTmfn5+fHn74YevtgIAAPfzwwzp16pS2bNni8M+iuD7++GM1adJE1157rc1rIad0J+e1kLNWX3zxhdd3i4D3IswC5VRSUpLS0tLUuHHjPPc1adJEFovFWof5+++/q0GDBnnG5XesMJ9++qlWrVql9evX6+DBg9q5c6fatm1rM6ZWrVp5Puzy+++/q2HDhnk6BeS8Bfv777/bHG/YsGGe527UqJFNbW9ycrLGjRuniIgIBQUFWUO2pHzrSq8+Z/369eXj41OmbZQOHDggwzDUsGFDaxjM+bVnzx6dOnVKklS3bl3FxcXp3XffVXh4uGJiYjR79uxC62VzDBo0SOnp6Vq2bJkk6eLFi/r66681YMAA69vyXbt21V133aWpU6cqPDxct99+uxYsWJCnVrMg0dHRWrVqlVavXq0ffvhBCQkJ+u9//2v9cGCOnPXI4chrNsfV6xYSEqIaNWrYrNuuXbt0xx13KCwsTKGhoapWrZr1A2pX/8xq1qypChUq2Bxr1KiRJJX5a2HXrl15Xgc5c8l5LQwaNEg33HCDHnzwQUVEROiee+7RkiVLCLbwKtTMAig1N910U57AcrXcV8mcaeDAgfrxxx/1j3/8Q61atVJISIgsFotuvfVWu/6hd0UrMYvFIpPJpG+++Ua+vr557g8JCbF+/eqrr2rEiBH64osvtHLlSo0dO9Za71u7du0Cn6Njx46Kjo7WkiVLNGTIEH355ZdKT0/XoEGDrGNMJpM++eQTbdq0SV9++aVWrFih+++/X6+++qo2bdpkM4/8VKhQQT169Cjy+y2L18K5c+fUtWtXhYaG6tlnn1X9+vUVGBiorVu36sknn3Tb0GexWNSiRQvNmDEj3/ujoqIkXfkZfvfdd1q3bp2++uorLV++XIsXL9bNN9+slStX5vs6AsobwixQTlWrVk3BwcHat29fnvv27t0rHx8f6z+IderU0cGDB/OMy++YM9SpU0e//PKLLBaLzdXZnLeA69SpYzP+wIEDec6xf/9+axeBs2fPas2aNZo6daomT55c6ONy35f7SuHBgwdlsVjs3s2rII6E4vr168swDNWtW9d6Ba4wLVq0UIsWLfTMM8/oxx9/1A033KC5c+fq+eefL/RxAwcO1Ouvv66UlBQtXrxY0dHR1hKM3Dp27KiOHTtq2rRpWrhwoYYOHapFixbpwQcftPt7coQjr9kcBw4cUPfu3a23L168qJMnT6pPnz6SruzSdubMGX322We66aabrONyOmxc7cSJE0pNTbW5Ort//35JKvPXwo4dO3TLLbcU+TgfHx/dcsstuuWWWzRjxgz985//1NNPP61169bZ9Z8KwNNRZgCUU76+vurVq5e++OILm7dHExMTtXDhQnXp0kWhoaGSpJiYGG3cuFHbt2+3jktOTtZHH31UJnPt06ePEhIStHjxYuuxy5cva9asWQoJCVHXrl1txn/++ec6fvy49fbmzZv1008/qXfv3pJkvRplGIbN42bOnFngHGbPnm1ze9asWZJkPWdx5dQLnzt3rsixd955p3x9fTV16tQ8czcMw9pWLCUlRZcvX7a5v0WLFvLx8bGrFGDQoEHKyMjQBx98oOXLl2vgwIE29589ezbP8+fUU9tbalAcjrxmc7zzzjs2db1z5szR5cuXC30tZGZm6q233sp3DpcvX9bbb79tM/btt99WtWrV8pTMOConINvzWhg4cKCOHz+uefPm5bkvPT1dqampkq78Ob1aWawV4E64Mgt4uPnz51v7c+Y2btw4Pf/889YelI899pj8/Pz09ttvKyMjQy+99JJ17Pjx4/Xhhx+qZ8+eGjNmjLU11zXXXKPk5GSnv+X+0EMP6e2339aIESO0ZcsWRUdH65NPPtGGDRs0c+ZMVaxY0WZ8gwYN1KVLFz366KPKyMjQzJkzVbVqVY0fP16SFBoaqptuukkvvfSSsrKyVKtWLa1cubLAq3HSlSt1/fr106233qqNGzfqww8/1JAhQ9SyZcsSfW9BQUFq2rSpFi9erEaNGqlKlSpq3ry5mjdvnmds/fr19fzzz2vixIk6cuSI+vfvr4oVK+rw4cNaunSpHnroIT3xxBNau3atYmNjNWDAADVq1EiXL1/Wv//9b/n6+uquu+4qck5t2rRRgwYN9PTTTysjI8OmxECSPvjgA7311lu64447VL9+fV24cEHz5s1TaGio9Yqns9j7ms2RmZmpW265RQMHDtS+ffv01ltvqUuXLurXr58kqXPnzqpcubKGDx+usWPHymQy6d///neesJ6jZs2aevHFF3XkyBE1atRIixcv1vbt2/XOO+84tCFGflq1aiVfX1+9+OKLOn/+vMxms26++WZVr149z9j77rtPS5Ys0SOPPKJ169bphhtuUHZ2tvbu3aslS5ZoxYoVateunZ599ll99913uu2221SnTh2dOnVKb731lmrXrm39QB9Q7rmukQKAkshpTVTQr2PHjhmGYRhbt241YmJijJCQECM4ONjo3r278eOPP+Y537Zt24wbb7zRMJvNRu3atY3p06cbb7zxhiHJSEhIKHQuOW2JimoP1LVrV6NZs2b53peYmGiMHDnSCA8PNwICAowWLVrYtD4yjL9aIr388svGq6++akRFRRlms9m48cYbbVpoGYZh/PHHH8Ydd9xhVKpUyQgLCzMGDBhgnDhxIk+brJy5796927j77ruNihUrGpUrVzZiY2ON9PR0m3MWpzWXYRjGjz/+aLRt29YICAiwef782jkZhmF8+umnRpcuXYwKFSoYFSpUMK699lpj9OjRxr59+wzDMIxDhw4Z999/v1G/fn0jMDDQqFKlitG9e3dj9erV+f5s8/P0008bkowGDRrkuW/r1q3G4MGDjWuuucYwm81G9erVjb/97W/Gzz//XOR5C1vj3CQZo0ePzvc+e16zOa//b7/91njooYeMypUrGyEhIcbQoUONM2fO2IzdsGGD0bFjRyMoKMioWbOmMX78eGPFihV51i5n7j///LPRqVMnIzAw0KhTp47x5ptv2pyvuK25DMMw5s2bZ9SrV8/w9fW1ef6rW3MZxpX2dC+++KLRrFkzw2w2G5UrVzbatm1rTJ061Th//rxhGIaxZs0a4/bbbzdq1qxpBAQEGDVr1jQGDx5s7N+/P9+fLVAemQyjgP+eAvB6f//73/X222/r4sWLfJAEAOCWqJkFIOlKHV5uZ86c0b///W916dKFIAsAcFvUzAKQJHXq1EndunVTkyZNlJiYqPfee08pKSmaNGmSq6cGAECBCLMAJF3pKPDJJ5/onXfekclkUps2bfTee+/ZtDMCAMDdUDMLAAAAj0XNLAAAADwWYRYAAAAey+tqZi0Wi06cOKGKFSu6ZO91AAAAFM4wDF24cEE1a9a02eY8P14XZk+cOJFnb28AAAC4n2PHjql27dqFjvG6MJuzLeaxY8fy7PEN18rKytLKlSvVq1evEm8bCffFOnsP1tp7sNbeoSzXOSUlRVFRUXm2M8+P14XZnNKC0NBQwqybycrKUnBwsEJDQ/nLsBxjnb0Ha+09WGvv4Ip1tqcklA+AAQAAwGMRZgEAAOCxCLMAAADwWIRZAAAAeCzCLAAAADwWYRYAAAAeizALAAAAj+XSMPvdd9+pb9++qlmzpkwmkz7//PMiH7N+/Xq1adNGZrNZDRo00Pvvv+/0eQIAAMA9uTTMpqamqmXLlpo9e7Zd4w8fPqzbbrtN3bt31/bt2/X3v/9dDz74oFasWOHkmQIAAMAduXQHsN69e6t37952j587d67q1q2rV199VZLUpEkT/fDDD3rttdcUExPjrGkCAAB4rWyLoc2Hk3XyXKoOnTcp22LInfZ586jtbDdu3KgePXrYHIuJidHf//73Ah+TkZGhjIwM6+2UlBRJV7Zky8rKcso8UTw568G6lG+ss/dgrb0Ha11+rdiVqOe/3quElJws5auPX/1Ok/pcq5hmEU57XkdeSx4VZhMSEhQRYfuDi4iIUEpKitLT0xUUFJTnMdOnT9fUqVPzHF+5cqWCg4OdNlcU36pVq1w9BZQB1tl7sNbeg7UuX3acMWn+/pyKVJP1eGLKJcUu2q77G1nUsqrhlOdOS0uze6xHhdnimDhxouLi4qy3U1JSFBUVpV69eik0NNSFM8PVsrKytGrVKvXs2VP+/u70BgZKE+vsPVhr78Falz/ZFkPTX/1OUkY+95pkkvRNYrDGD71Jvj6mfMaUTM476fbwqDAbGRmpxMREm2OJiYkKDQ3N96qsJJnNZpnN5jzH/f39+QPnplgb78A6ew/W2nuw1uXHz7+dyVVakJch6eT5DG3744I61a9a6s/vyOvIo/rMdurUSWvWrLE5tmrVKnXq1MlFMwIAACh/Tl24VKrjnMmlYfbixYvavn27tm/fLulK663t27fr6NGjkq6UCAwbNsw6/pFHHtGhQ4c0fvx47d27V2+99ZaWLFmixx9/3BXTBwAAKJeOnE61a1z1ioFOnknRXBpmf/75Z7Vu3VqtW7eWJMXFxal169aaPHmyJOnkyZPWYCtJdevW1VdffaVVq1apZcuWevXVV/Xuu+/SlgsAAKCUZFsM/Wfz0SLH1QgLVPu6VcpgRoVzac1st27dZBgFfwouv929unXrpm3btjlxVgAAAN5r8+HkQutlc9xz/TVO+fCXozyqZhYAAADOtXp3gl3josPdo8UpYRYAAACSrpQYLN1+3K6x7lAvKxFmAQAA8KfNh5OVnFr07ltVKwS4Rb2sRJgFAADAn+wtMbi9VU23qJeVCLMAAACQtHznSb234YhdY3s2jXTuZBxAmAUAAPBy2RZDU7/cbcdIQzXCzG5TYiARZgEAALze5sPJOnnevt28nu59rduUGEiEWQAAAK9n77a0XWtYFNMswsmzcQxhFgAAwMvZu31ti8pOnkgxEGYBAAC8mL3b10aGmlU/tOCdW12FMAsAAODF7N2+dlC72nKjUlkrwiwAAIAXs7e3bJ2q7rF97dUIswAAAF7Kse1rzU6eTfEQZgEAALyUI9vXtqvjhp/+EmEWAADAa3ni9rVXI8wCAAB4IU/dvvZqhFkAAAAvY//2tVKNsEC32r72aoRZAAAAL+PI9rXxfZu6bYmBRJgFAADwOvbWyt5/Q7RubV7DybMpGcIsAACAF3GkHZc718rmIMwCAAB4EUfacblzrWwOwiwAAIAXOXXBvlpZd27HlRthFgAAwIscOZ1q1zhPKDGQCLMAAABeI9tiaOFPR4sc5+7tuHIjzAIAAHiJN9ceUOKFjCLH3XP9NR5RYiARZgEAALzC8p0n9drqA3aNjQ4PdvJsSg9hFgAAoJxzZMcvSapeMdCJsyldhFkAAIByzpEdvzypXlYizAIAAJR79rbjktx/+9qrEWYBAADKOXvbcT3eo5Hbb197NcIsAABAOZZtMfSfzUW344oMNSv25gZlMKPSRZgFAAAoxzYfTlZCStHtuAa3r+NR5QU5CLMAAADlmL31sp7Ujis3wiwAAEA5Zm+9rCe148qNMAsAAFBOlcfta69GmAUAACinyuP2tVcjzAIAAJRD5XX72qsRZgEAAMqZ8rx97dUIswAAAOVMed6+9mqEWQAAgHKmPG9fezXCLAAAQDlTnrevvRphFgAAoByxtx2Xp25fezXCLAAAQDlibzsuT92+9mqEWQAAgHLCW9px5UaYBQAAKAe8qR1XboRZAACAcsCb2nHlRpgFAAAoB7ypHVduhFkAAIBywJvaceVGmAUAAPBw3taOKzfCLAAAgIfztnZcuRFmAQAAPJg3tuPKjTALAADgoby1HVduhFkAAAAP5a3tuHIjzAIAAHgob23HlRthFgAAwEN5azuu3AizAAAAHijbYug/m72zHVduhFkAAAAPtPlwshJSvLMdV26EWQAAAA+0eneCXePKYzuu3AizAAAAHmb5zpN6b8MRu8aWx3Zcufm5egIAAACwn729ZU2SIstpO67cXH5ldvbs2YqOjlZgYKA6dOigzZs3Fzp+5syZaty4sYKCghQVFaXHH39cly7Z35YCAADAk9nbW9ZQ+W3HlZtLw+zixYsVFxen+Ph4bd26VS1btlRMTIxOnTqV7/iFCxdqwoQJio+P1549e/Tee+9p8eLFeuqpp8p45gAAAK5hb63s/TdEl9t2XLm5NMzOmDFDo0aN0siRI9W0aVPNnTtXwcHBmj9/fr7jf/zxR91www0aMmSIoqOj1atXLw0ePLjIq7kAAADlQbbF0NLtx+0a27NppJNn4x5cVjObmZmpLVu2aOLEidZjPj4+6tGjhzZu3JjvYzp37qwPP/xQmzdvVvv27XXo0CF9/fXXuu+++wp8noyMDGVk/NW2IiUlRZKUlZWlrKysUvpuUBpy1oN1Kd9YZ+/BWnsP1rrs/HQ4WcmpRf+cq1TwV+vaFUt1TcpynR15DpeF2dOnTys7O1sRERE2xyMiIrR37958HzNkyBCdPn1aXbp0kWEYunz5sh555JFCywymT5+uqVOn5jm+cuVKBQeX71YVnmrVqlWungLKAOvsPVhr78FaO99nh31kzxvrLUIvacXyb5wyh7JY57S0NLvHelQ3g/Xr1+uf//yn3nrrLXXo0EEHDx7UuHHj9Nxzz2nSpEn5PmbixImKi4uz3k5JSVFUVJR69eql0NDQspo67JCVlaVVq1apZ8+e8vf3d/V04CSss/dgrb0Ha102VuxK1Lcbd9g1dlTvDupQyl0MynKdc95Jt4fLwmx4eLh8fX2VmJhoczwxMVGRkfnXeEyaNEn33XefHnzwQUlSixYtlJqaqoceekhPP/20fHzy/k/FbDbLbDbnOe7v788fODfF2ngH1tl7sNbeg7V2nmyLoWnf7LNrbI2wQHVqUN1pXQzKYp0dOb/LPgAWEBCgtm3bas2aNdZjFotFa9asUadOnfJ9TFpaWp7A6uvrK0kyDMN5kwUAAHAhe9txSd7Rjis3l5YZxMXFafjw4WrXrp3at2+vmTNnKjU1VSNHjpQkDRs2TLVq1dL06dMlSX379tWMGTPUunVra5nBpEmT1LdvX2uoBQAAKG9ox1Uwl4bZQYMGKSkpSZMnT1ZCQoJatWql5cuXWz8UdvToUZsrsc8884xMJpOeeeYZHT9+XNWqVVPfvn01bdo0V30LAAAATkU7rsK5/ANgsbGxio2Nzfe+9evX29z28/NTfHy84uPjy2BmAAAArrfZznZcVSsElPuta/Pj8u1sAQAAUDB7Swxub1XTq2plcxBmAQAA3NTynSf13oYjdo31xhIDiTALAADglrIthqZ+uduusTXCAr2yxEAizAIAALgl2nHZhzALAADghmjHZR/CLAAAgJuhHZf9CLMAAABuhnZc9iPMAgAAuBnacdmPMAsAAOBGaMflGMIsAACAm6Adl+MIswAAAG6CdlyOI8wCAAC4CdpxOY4wCwAA4AaolS0eP1dPAAAAwNvZWytrkhRJrawNrswCAAC4mL21soaolb0aYRYAAMDFqJUtPsIsAACAC7F1bckQZgEAAFyIrWtLhjALAADgQmxdWzKEWQAAABehHVfJEWYBAABcINtiaMoytq4tKcIsAACAC7y59oASUti6tqQIswAAAGVs+c6Tem31AbvG0o6rcIRZAACAMmTvbl85qJUtHGEWAACgDNm725dEraw9CLMAAABl6NQF+4KsRK2sPQizAAAAZejI6VS7xj3eoxG1snYgzAIAAJSRbIuhhT8dLXJcZKhZsTc3KIMZeT7CLAAAQBl5c+0BJV7IKHLc4PZ1KC+wE2EWAACgDDjSjis6PNjJsyk/CLMAAABO5mg7ruoVA504m/KFMAsAAOBktONyHsIsAACAk63enWD3WNpxOYYwCwAA4ETLd57UexuO2DWWdlyOI8wCAAA4iSO1srTjKh7CLAAAgJM4Uis7pV8zyguKgTALAADgJPbWyt5/QzTlBcVEmAUAAHACR2plezaNdO5kyjE/V08AAACgvMm2GJqyrOhaWZOkSFpxlQhXZgEAAErZm2sPKCGl6FpZQ7TiKinCLAAAQClyZNtaamVLjjALAABQShzdtpZa2ZIjzAIAAJQStq0te4RZAACAUsK2tWWPMAsAAFAK2LbWNQizAAAAJcS2ta5DmAUAACghtq11HcIsAABACbFtresQZgEAAEog22Jo6fbjdo2lFVfpI8wCAACUwObDyUpOzSpyXNUKAbTicgLCLAAAQAnYW2Jwe6ua1Mo6AWEWAACgmBxpx0WJgXMQZgEAAIoh22JoyjL72nGx25fzEGYBAACK4c21B5SQYl87Lnb7ch4/RwafO3dOS5cu1ffff6/ff/9daWlpqlatmlq3bq2YmBh17tzZWfMEAABwG8t3ntRrqw/YNZZ2XM5l15XZEydO6MEHH1SNGjX0/PPPKz09Xa1atdItt9yi2rVra926derZs6eaNm2qxYsXO3vOAAAALuPIbl8StbLOZteV2datW2v48OHasmWLmjZtmu+Y9PR0ff7555o5c6aOHTumJ554olQnCgAA4A4c2e2LWlnnsyvM7t69W1WrVi10TFBQkAYPHqzBgwfrzJkzpTI5AAAAd2NvKy6JWtmyYFeZQVFBtqTjAQAAPIEjrbge79GIWtky4NAHwHLs27dPs2bN0p49eyRJTZo00ZgxY9S4ceNSnRwAAIC7cKRWNjLUrNibGzh5RpCK0Zrr008/VfPmzbVlyxa1bNlSLVu21NatW9W8eXN9+umnzpgjAACAyzlSKzulXzPKC8qIw2F2/PjxmjhxojZu3KgZM2ZoxowZ+vHHH/XUU09p/PjxDk9g9uzZio6OVmBgoDp06KDNmzcXOv7cuXMaPXq0atSoIbPZrEaNGunrr792+HkBAAAcYW+tLK24ypbDYfbkyZMaNmxYnuP33nuvTp486dC5Fi9erLi4OMXHx2vr1q1q2bKlYmJidOrUqXzHZ2ZmqmfPnjpy5Ig++eQT7du3T/PmzVOtWrUc/TYAAADsxra17svhmtlu3brp+++/V4MGtnUgP/zwg2688UaHzjVjxgyNGjVKI0eOlCTNnTtXX331lebPn68JEybkGT9//nwlJyfrxx9/lL+/vyQpOjra0W8BAADAbvZuW2uSFEkrrjLncJjt16+fnnzySW3ZskUdO3aUJG3atEkff/yxpk6dqmXLltmMLUhmZqa2bNmiiRMnWo/5+PioR48e2rhxY76PWbZsmTp16qTRo0friy++ULVq1TRkyBA9+eST8vX1zfcxGRkZysjIsN5OSUmRJGVlZSkrK8v+bxxOl7MerEv5xjp7D9bae5T3tZ619je7tq01JD3du7Es2ZdlyXb+vMpaWa6zI89hMgzDcOTkPj72VSaYTCZlZxe8kidOnFCtWrX0448/qlOnTtbj48eP17fffquffvopz2OuvfZaHTlyREOHDtVjjz2mgwcP6rHHHtPYsWMVHx+f7/NMmTJFU6dOzXN84cKFCg4Otut7AQAA3mnHGZPm7/fRleuuhetaI1t3RjsUq1CAtLQ0DRkyROfPn1doaGihYx2+MmuxWIo9sZKyWCyqXr263nnnHfn6+qpt27Y6fvy4Xn755QLD7MSJExUXF2e9nZKSoqioKPXq1avIHw7KVlZWllatWqWePXtay0hQ/rDO3oO19h7lda2zLYamv/qdpIwix0rSqN4d1KEclxiU5TrnvJNuj2L1mS0N4eHh8vX1VWJios3xxMRERUbmXzhdo0YN+fv725QUNGnSRAkJCcrMzFRAQECex5jNZpnN5jzH/f39y9UfuPKEtfEOrLP3YK29R3lb659/O6OEFPuCbI2wQHVqUN0r2nGVxTo7cn67agYWLVpk9wmPHTumDRs2FDkuICBAbdu21Zo1a6zHLBaL1qxZY1N2kNsNN9yggwcP2lwd3r9/v2rUqJFvkAUAACgutq31DHaF2Tlz5qhJkyZ66aWXrLt+5Xb+/Hl9/fXXGjJkiNq0aaMzZ87Y9eRxcXGaN2+ePvjgA+3Zs0ePPvqoUlNTrd0Nhg0bZvMBsUcffVTJyckaN26c9u/fr6+++kr//Oc/NXr0aLueDwAAwB5sW+s57Coz+Pbbb7Vs2TLNmjVLEydOVIUKFRQREaHAwECdPXtWCQkJCg8P14gRI7Rz505FRETY9eSDBg1SUlKSJk+erISEBLVq1UrLly+3Pv7o0aM2HziLiorSihUr9Pjjj+u6665TrVq1NG7cOD355JPF+NYBAADyyrxs0VNLd9o1lm1rXc/umtl+/fqpX79+On36tH744Qf9/vvvSk9PV3h4uFq3bq3WrVvb3ekgt9jYWMXGxuZ73/r16/Mc69SpkzZt2uTw8wAAABRl+c6Temrpr0pOta81FNvWup7DHwALDw9X//79nTAVAAAA11m+86Qe/XCr7G2uxba17sHxS6kAAADlTLbF0NQvd9sdZCW2rXUXLmvNBQAA4C42H07WyfNF7/IlsW2tu+HKLAAA8HqOtOGSaMXlTgizAADAqznShqtqhQDNubcNtbJuxOEwu27dOmfMAwAAoMxlWwxNWbbbrrFVKvhr48RbCLJuxuEwe+utt6p+/fp6/vnndezYMWfMCQAAoEy8ufaAElLsq5X95x0tFODHm9ruxuEVOX78uGJjY/XJJ5+oXr16iomJ0ZIlS5SZmemM+QEAADjF8p0n9drqA3aNpQ2X+3I4zIaHh+vxxx/X9u3b9dNPP6lRo0Z67LHHVLNmTY0dO1Y7duxwxjwBAABKTU4rLnvRhst9lehaeZs2bTRx4kTFxsbq4sWLmj9/vtq2basbb7xRu3btKq05AgAAlCpHWnHVoA2XWytWmM3KytInn3yiPn36qE6dOlqxYoXefPNNJSYm6uDBg6pTp44GDBhQ2nMFAAAoFY604qINl3tzeNOEMWPG6D//+Y8Mw9B9992nl156Sc2bN7feX6FCBb3yyiuqWbNmqU4UAACgNDjSiuvxHo2olXVzDofZ3bt3a9asWbrzzjtlNpvzHRMeHk4LLwAA4HYcqZWNDDUr9uYGTp4RSsrhMoP4+HgNGDAgT5C9fPmyvvvuO0mSn5+funbtWjozBAAAKCWbDp2xu1Z2Sr9mlBd4AIfDbPfu3ZWcnJzn+Pnz59W9e/dSmRQAAEBpW77zpEZ/tNWusbTi8hwOlxkYhiGTKe//Us6cOaMKFSqUyqQAAABK0/KdJ/Xoh1tl2DmeVlyew+4we+edd0qSTCaTRowYYVNmkJ2drV9++UWdO3cu/RkCAACUQM6WtfYEWZOkSFpxeRS7w2xYWJikK1dmK1asqKCgIOt9AQEB6tixo0aNGlX6MwQAACgBR7aslWjF5WnsDrMLFiyQJEVHR+uJJ56gpAAAALg9R7asrRTsrxfubEGtrIdxuGY2Pj7eGfMAAAAoVY5uWTt7cBvd0DDciTOCM9gVZtu0aaM1a9aocuXKat26db4fAMuxdat9nxIEAABwJke3rO1Yv6qTZwRnsCvM3n777dYPfPXv39+Z8wEAACgVbFnrHewKs7lLCygzAAAA7o4ta72Hw5smAAAAuLPMyxY9tXSnXWPZstbz2XVltnLlyoXWyeaW3+5gAAAAZWH5zpN6aumvSk7Nsms8W9Z6PrvC7MyZM508DQAAgJJxdJcvtqwtH+wKs8OHD3f2PAAAAIotpw2XvUFWYsva8sKuMJuSkqLQ0FDr14XJGQcAAFBWHGnDxZa15YvdNbMnT55U9erVValSpXzrZw3DkMlkUnZ2dqlPEgAAoDCOtOGSaMVVntgVZteuXasqVa7872XdunVOnRAAAIAjHGnDVbVCgKbd0Zxa2XLErjDbtWvXfL8GAABwJUfacFWp4K+NE29RgB+dScsTu8Ls1c6ePav33ntPe/bskSQ1bdpUI0eOtF69BQAAcDZH23D9844WBNlyyOEV/e677xQdHa033nhDZ8+e1dmzZ/XGG2+obt26+u6775wxRwAAABs5bbjsDbK04Sq/HL4yO3r0aA0aNEhz5syRr6+vJCk7O1uPPfaYRo8erV9//bXUJwkAAJCDNlzIzeErswcPHtT//d//WYOsJPn6+iouLk4HDx4s1ckBAABczdE2XDVow1WuORxm27RpY62VzW3Pnj1q2bJlqUwKAACgILThQm52lRn88ssv1q/Hjh2rcePG6eDBg+rYsaMkadOmTZo9e7ZeeOEF58wSAABAtOFCXnaF2VatWslkMskw/qpOGT9+fJ5xQ4YM0aBBg0pvdgAAAH/KqZW1B224vIddYfbw4cPOngcAAEChNh06Y3etLG24vIddYbZOnTrOngcAAECBlu88qQmf2tcxiTZc3qVYmyZI0u7du3X06FFlZmbaHO/Xr1+JJwUAAJAjp6esva24aMPlXRwOs4cOHdIdd9yhX3/91aaO1mS68inB7Ozs0p0hAADwWjnb1doTZE2SImnD5XUcLiYZN26c6tatq1OnTik4OFi7du3Sd999p3bt2mn9+vVOmCIAAPBGy3eeVMfpq5Wcmln04D/Rhsv7OHxlduPGjVq7dq3Cw8Pl4+MjHx8fdenSRdOnT9fYsWO1bds2Z8wTAAB4EUdLCyoF++uFO1tQK+uFHL4ym52drYoVK0qSwsPDdeLECUlXPiS2b9++0p0dAADwOsXZrnb24DYEWS/l8JXZ5s2ba8eOHapbt646dOigl156SQEBAXrnnXdUr149Z8wRAAB4EUe3q40MC1TH+lWdOym4LYfD7DPPPKPU1FRJ0rPPPqu//e1vuvHGG1W1alUtXry41CcIAAC8C9vVwhEOh9mYmBjr1w0aNNDevXuVnJysypUrWzsaAAAAFAfb1cJRxe4zK0nHjh2TJEVFRZXKZAAAgPfKacNlD7arRQ6HXwGXL1/WpEmTFBYWpujoaEVHRyssLEzPPPOMsrKynDFHAABQzjnahovtapHD4SuzY8aM0WeffaaXXnpJnTp1knSlXdeUKVN05swZzZkzp9QnCQAAyi9H23CxXS1yczjMLly4UIsWLVLv3r2tx6677jpFRUVp8ODBhFkAAGC34rThYrta5OZwmDWbzYqOjs5zvG7dugoICCiNOQEAAC+x6dAZh9twsV0tcnO42CQ2NlbPPfecMjIyrMcyMjI0bdo0xcbGlurkAABA+bV850mN/mirQ4+hDReuZteV2TvvvNPm9urVq1W7dm21bNlSkrRjxw5lZmbqlltuKf0ZAgCAcsfROlnacKEgdoXZsLAwm9t33XWXzW1acwEAAHvltOCyN8jShguFsSvMLliwwNnzAAAAXmD5zpN6aumvSk61r52nSbThQuGKvWlCUlKS9u3bJ0lq3LixqlWrVmqTAgAA5Y+jpQWVgv31wp0tKC1AoRz+b05qaqruv/9+1ahRQzfddJNuuukm1axZUw888IDS0tKcMUcAAODhitOCa/bgNgRZFMnhMBsXF6dvv/1WX375pc6dO6dz587piy++0Lfffqv/+7//c8YcAQCAh3O0BVeNsEB1rF/VuZNCueBwmP3000/13nvvqXfv3goNDVVoaKj69OmjefPm6ZNPPinWJGbPnq3o6GgFBgaqQ4cO2rx5s12PW7RokUwmk/r371+s5wUAAM5HCy44k8NhNi0tTREREXmOV69evVhlBosXL1ZcXJzi4+O1detWtWzZUjExMTp16lShjzty5IieeOIJ3XjjjQ4/JwAAKBs5dbLn0u37wFfVCgGacy/lBbCfw2G2U6dOio+P16VLf71VkJ6erqlTp6pTp04OT2DGjBkaNWqURo4cqaZNm2ru3LkKDg7W/PnzC3xMdna2hg4dqqlTp6pevXoOPycAAHA+R+tkc1pwEWThCIe7GcycOVO33nprnk0TAgMDtWLFCofOlZmZqS1btmjixInWYz4+PurRo4c2btxY4OOeffZZVa9eXQ888IC+//77Qp8jIyPDZreylJQUSVJWVpaysuz7XyLKRs56sC7lG+vsPVhr71HQWv/4m2N1ss/2bSqTka2srOzSniJKQVn+mXbkORwOsy1atNCBAwf00Ucfae/evZKkwYMHa+jQoQoKCnLoXKdPn1Z2dnaesoWIiAjrua/2ww8/6L333tP27dvteo7p06dr6tSpeY6vXLlSwcHBDs0XZWPVqlWungLKAOvsPVhr75F7rXecMWnRbz66ElMLF+xn6J56FmX/vkVf/+7ECaJUlMWfaUdKVx0Ks1lZWbr22mv13//+V6NGjXJ4YiV14cIF3XfffZo3b57Cw8PteszEiRMVFxdnvZ2SkqKoqCj16tVLoaGhzpoqiiErK0urVq1Sz5495e/v7+rpwElYZ+/BWnuPq9d6xa5ELdi4w+7ygrn3tlMnOhe4vbL8M53zTro9HAqz/v7+NrWyJRUeHi5fX18lJibaHE9MTFRkZGSe8b/99puOHDmivn37Wo9ZLBZJkp+fn/bt26f69evbPMZsNstsNuc5l7+/P3+5uinWxjuwzt6DtfYe/v7+Mky+mvzlHruCrElSZFigbmgUQecCD1IWf6YdOb/DHwAbPXq0XnzxRV2+fNnRh+YREBCgtm3bas2aNdZjFotFa9asyffDZNdee61+/fVXbd++3fqrX79+6t69u7Zv366oqKgSzwkAABTPil2J6jh9tZJTM+1+DC24UFIO18z+73//05o1a7Ry5Uq1aNFCFSpUsLn/s88+c+h8cXFxGj58uNq1a6f27dtr5syZSk1N1ciRIyVJw4YNU61atTR9+nQFBgaqefPmNo+vVKmSJOU5DgAAys6OMyaHSgvYqhalxeEwW6lSJd11112lNoFBgwYpKSlJkydPVkJCglq1aqXly5dbPxR29OhR+fg4fAEZAACUkWyLoc+O+Di8Ve0NDe37/AtQGIfD7IIFC0p9ErGxsYqNjc33vvXr1xf62Pfff7/U5wMAAOyTbTH0r02/61ymfaUCOXWybFWL0mJ3mLVYLHr55Ze1bNkyZWZm6pZbblF8fLzD7bgAAED5sHznSU39crfdvWRzUCeL0mT3+/fTpk3TU089pZCQENWqVUuvv/66Ro8e7cy5AQAAN5WzTa0jQZatauEMdl+Z/de//qW33npLDz/8sCRp9erVuu222/Tuu+9S0woAgBfJvGzRU0t3OlQjm7NVbYAfmQGly+5X1NGjR9WnTx/r7R49eshkMunEiRNOmRgAAHA/y3eedKj9lunPX/+8owVBFk5h95XZy5cvKzAw0OaYv78/e24DAOAlckoLHLkiGxkWqPi+TSktgNPYHWYNw9CIESNsdtO6dOmSHnnkEZtes472mQUAAO4v22Jo6pe7HQqyk25rohE31OXDXnAqu8Ps8OHD8xy79957S3UyAADAPW06dMbuD3vltN8iyKIs2B1mndFfFgAAuL/lO09qwqe/OvQY2m+hrDi8aQIAAPAejtbJVq0QoGl3NKdGFmWGMAsAAPLIthja9NsZTfj0VzuDrKEqFQJov4UyR5gFAAA2iruz17N9mxJkUeYIswAAwKo47bcqBfnrzqhLimkW4bR5AQXhv08AAEBS8dpvSdIbg65Ty6qOPgooHYRZAACgbIuh9zccdqi0wCSpRlig2tet4ryJAUWgzAAAAC9XnBrZnKZbtOCCqxFmAQDwYsWpkZVst6lla3u4EmEWAAAvVZwa2UpB/po9tI061qvKFVm4BcIsAABeytEtaiXphbta6IYG4c6bFOAgwiwAAF7I0S1qc5cVAO6EMAsAgJdxtE520m1NNOKGupQVwC0RZgEA8BKOblFr0pUrsgRZuDPCLAAAXqC4W9TSegvujjALAEA5V6wtaoP99cKdLaiRhdsjzAIAUI4Vd4va2YPb6IaGdC2A+yPMAgBQThV3i9rIsEB1rF/VeRMDShFhFgCAcogtauEtCLMAAJQzpbFFLeApCLMAAJQD2RZDmw8nK+F8up77ag9b1MJrEGYBAPBwxW27xRa1KA8IswAAeLDilhRIlBWgfCDMAgDgoYrbdktii1qUH4RZAAA8UHHabklsUYvyhzALAICHKWmNLK23UJ4QZgEA8CDUyAK2CLMAAHiAbIuhTb+d0YRPf3UoyFap4K9Jf2umyNBAta9bhSuyKHcIswAAuLmS7Ob1zztacCUW5RphFgAAN8ZuXkDhCLMAALip4rbeou0WvAlhFgAAN5OzNe2Gg0kOlxbQdgvehjALAIAboe0W4BjCLAAAboK2W4DjCLMAALiB4tbHVgry1+yhbdSxXlWuyMIrEWYBAHChktTHStILd7XQDQ3CnTM5wAMQZgEAcJHi1sdKlBUAOQizAAC4QHHrY2O7N9ANDcLZzQv4E2EWAIAyklNSkHA+Xc99tcehIJvTduvxno0IsUAuhFkAAMpASUoKaLsFFIwwCwCAk5Wk5ZZEfSxQGMIsAABOVNyWWxL1sYA9CLMAADhBcVtuSdTHAo4gzAIAUMqojwXKDmEWAIBSRH0sULYIswAAlFBJWm5JUpUK/pr0t2aKDA2kPhZwEGEWAIASKI2Sgn/e0YIrsUAxEWYBACgmSgoA1yPMAgBQDLTcAtwDYRYAADvl1MaeunBJpy9k0HILcAOEWQAA7FCS2liJlluAsxBmAQAoQklrYyXqYwFnIcwCAHCV3OUE4RXMmrKseLWxtNwCnM/H1ROQpNmzZys6OlqBgYHq0KGDNm/eXODYefPm6cYbb1TlypVVuXJl9ejRo9DxAAA4YvnOk+ry4loNnrdJ4xZt19D3flJCiuO1sSZdabl1R+ta6lS/KkEWcBKXh9nFixcrLi5O8fHx2rp1q1q2bKmYmBidOnUq3/Hr16/X4MGDtW7dOm3cuFFRUVHq1auXjh8/XsYzBwCUNznlBMWti80RGRaoOfe2oaQAKAMuLzOYMWOGRo0apZEjR0qS5s6dq6+++krz58/XhAkT8oz/6KOPbG6/++67+vTTT7VmzRoNGzasTOYMACh/StJqS5Im3dZE4RXNql6RkgKgLLk0zGZmZmrLli2aOHGi9ZiPj4969OihjRs32nWOtLQ0ZWVlqUqVKvnen5GRoYyMDOvtlJQUSVJWVpaysrJKMHuUtpz1YF3KN9bZe3jCWmdbDP38+1mdupCh0xcdb7Ul5bTbMmto+9rWAGvJvixLdilP1o15wlqj5MpynR15DpeG2dOnTys7O1sRERE2xyMiIrR37167zvHkk0+qZs2a6tGjR773T58+XVOnTs1zfOXKlQoODnZ80nC6VatWuXoKKAOss/dw17Xeccakz4746FxmSa6gGjIk9Y5I04rl35TW1DyWu641SldZrHNaWprdY11eZlASL7zwghYtWqT169crMDAw3zETJ05UXFyc9XZKSoq1zjY0NLSspgo7ZGVladWqVerZs6f8/f1dPR04CevsPdx5rVfsStSCjTtK1GpLkmqEBerp3tcqpllE0YPLMXdea5SeslznnHfS7eHSMBseHi5fX18lJibaHE9MTFRkZGShj33llVf0wgsvaPXq1bruuusKHGc2m2U2m/Mc9/f35w+cm2JtvAPr7D3cba2zLYamfbPP4SBrkhQRatarA1vp9MUMamPz4W5rDecoi3V25PwuDbMBAQFq27at1qxZo/79+0uSLBaL1qxZo9jY2AIf99JLL2natGlasWKF2rVrV0azBQB4otw9Y6tXDJTFMIq1Da0kTenXTDc0CC/9SQIoNpeXGcTFxWn48OFq166d2rdvr5kzZyo1NdXa3WDYsGGqVauWpk+fLkl68cUXNXnyZC1cuFDR0dFKSEiQJIWEhCgkJMRl3wcAwP3ktwVtaKDj//SxexfgvlweZgcNGqSkpCRNnjxZCQkJatWqlZYvX279UNjRo0fl4/NXO9w5c+YoMzNTd999t8154uPjNWXKlLKcOgDAjRW0BW3Kpct2PZ5WW4BncHmYlaTY2NgCywrWr19vc/vIkSPOnxAAwKNcXUrQtk7lYveMvdJqK1AjbqhLgAU8gFuEWQAAiiu/UoIqFfyVnOp4L8yc6BrftylBFvAQhFkAgMcqqJTA3iBbKchf59L/GkttLOB5CLMAAI9U0u1nJWn2kDby8TFZyxOojQU8D2EWAOARSqPFVo6cutiO9asSXgEPR5gFALi9/Opiw4KK908YdbFA+UKYBQC4tYLqYs+n29diq0qFACWnZlpvUxcLlC+EWQCAW7i6jKB93SqSVOIWW9/+o7u2/H6WulignCLMAgBcLr8yghphgbrn+qhi1cXmLiUI8PNRp/pVS2mmANwNYRYA4FIFlRGcPH9Jr60+YNc5aLEFeC/CLADAZUqjvZZEiy3AmxFmAQBOl189rK+PSZsPJxe7vZZEiy0AhFkAgJOt2JWoad/sy1MPG9+3qQ6fTrX7PCbJ5gouLbYASIRZAIAT7Thj0oKNO/Kth33kw612n+fxHo206H9HbQIxdbEAJMIsAKCECiohyLYY+uyIT5H1sGY/H2VctuR7X04ZQezNDRR7c4N8nweAdyPMAgCKraCWWvF9myokwEfnMosOm491a6CZq/dLKrqMgBZbAK7m4+oJAAA8U05Lras/wJXwZwnBKyv323We6PBgzbm3jSLDAm2OR4YFas69bSgjAFAorswCABxWWEutnGPb/0ix61zVKwaqU/2q6tk0kjICAA4jzAIA8lVQLawku1tqmX0MZVpM+YbenHrYnG1rfX1MlBEAcBhhFgCQR2G1sLc2r6GDpy7YdZ6OERZ9d9KXtloAnIYwCwCwUdD2sjm1sE1rhGrPSftKCFpUlgZ0bZmnzyxttQCUFsIsAMDKnlrY3X8GWX9fk7Ky82+8daWEwKz6oamKaRah3tfVoh4WgFMQZgHAy5RGLexrA1sqKMBXj/658UF+JQRP975W2b9vkUQ9LADnIcwCgBcpqBZ20m1NFV7RrLfWH7TrPD4+Jt3avIbm3Nsmz/lySghuaRyur38v9W8BAGwQZgHASxRUC3vy/CU9ttD+rWWlK+20JOnW5jUKbKmVlZVVSjMHgIIRZgHAwxVWNpB7TEG1sDlMkvq3qqn1+5N0Li3LrnZaEiUEAFyLMAsAHqyoFlqSlHIpS+9+f6jIWlhD0sDrr1FM80g9+uFW2mkB8AiEWQDwUEW10BrULkonzqdr06EzBXYduNqpC5d0e6tahdbC0k4LgDshzAKAB7Knhdbin49Zj9UIC7SrS4E9tbAA4E4IswDgRuypf5Xsb6E1tEOU7u9ST9FVK6jLi2uVcP4StbAAyhXCLAC4iaLqX7OyLdp29Jy+25+kL3Yct+uc7etWVf1qIZKu1LpSCwugvCHMAoAbKKxt1iMfblXL2mE6lJSqCxmXHTpvTtmApCL7wlILC8ATEWYBwAnsLRfIGVtU26wdf5yXJFWpEKAuDcLVpWG4XlmxT0kXMuwuG5CohQVQ/hBmAaCU2dMuK0dyaqb+vfGIXfWv0/o31+D218jnz+AZGuhXrLIBamEBlCeEWQAoRYW1y3r0w6168e7rVCHATz8dPqOfDiVrX+IFu88dEuhnDbISZQMAIBFmAaDU2NMua/wnv+S5r3blQP1x1v62WblRNgDA2xFmAeAqjtS75rbxt9N2lQtEVQ7SLU0i1KFuFbWvW0WVggMcbpuVG2UDALwZYRYAcnGk3vVUyiVtPXpO24+d07ajZ7X16Fm7nuOJmMa6vVUtm2O0zQKA4iHMAsCfiqp3fSKmkcx+vtp27Jy2Hz2n4+fSi/U8BZULUP8KAI4jzAIoF4pbGpD78UXVu768Yr/NcZNJahxRUa2vqaTWUZV1Xe0wjVjwPyWmFK9cgPpXAHAcYRaAx3OkNOBql7KytT/xgr7cccKuetc211TWLU2qq/U1lXRd7UoKMdv+NTqlX8nKBah/BQDHEGYBeLSiSgPm3NvGGmjPpWVq94kU7T6Zol0nUrT7RIoOJl1UtqWw7QpsDe9cJ0+9a26UCwBA2SLMAvBY9pQGPPHxL/r45z+0N+FCgTWulYP9VatSkHaeSCnyOfOrd70a5QIAUHYIswDKVLbF0IHzJn35y0nVqFShRCFv8+EzRZYGXMy4rDV7T1lvX1MlWE1rhKpZzVA1/fNXZGigLIZK1B7rapQLAEDZIMwCKDPLd57UlGW7lJDiK+3+VZJ9ta3ZFkN/nE3TwVMX//qVdFF7TxZ9JVWS7mxTS4PaRalJzVCFBvrnO8bXRHssAPBEhFkAZcKe2tZujavr8OlUm8D626mLOnQ6VZmXLcV+7gFto9ShXtFXSal3BQDPQ5gF4HT21LaOXrhNFouR7xhJMvv5qF61ENWvVkENqoeoQfUQ1Q2voAfe/58SUzJKpTRAot4VADwNYRZAiXu0Xu1SVrb+OJuuY2fT9EdymjYdKrq2NaejQGignzWsWn9Vq6halYPyndOUfs1KvTSAelcA8ByEWcDLFadHq8ViKPHCJR09k6ZjZ9N1NPlKaD12Nk1Hk9OUmJJRrLk817+Z7u1QRyaT/eGT0gAA8G6EWcCLFVXHOqH3tYqqEqyjyWk6lnwluB5LTtPxs+nKzC68hrVCgK+iqgQrqkqw/HxM+mZnQpHzaVCtokNBNgelAQDgvQizgAuU9tv6jkrPzNbJ8+l6eunOQutYp3+zt8Bz+PmYVLNSkK6pEqyoKkGqXTn4z6+v/F452N8aTLMtRqm2vcoPpQEA4J0Is0AZK8nWq0XJuJytUykZOnXhkhJTMnQq5ZISL2QoMeWSTqVc+T0x5ZJSLl22+5wNq1dQ05phivozrNauciXARoYGys/Xx65z+PqYaHsFAHAKwixQhhzZejW3rGyLTl/MUGJKTjC9ZP068cKfoTXlks6mZdk9F39fk7Kyi97GNfbmhoVu32qvnNrWK31m/6qppbYVAFAShFmgjNjTnurJT3/VzhMpOn0h5yrqlausZ1IzZRSdOyVJAX4+igg1q3rFwFy/X/k65/fqoYHadfy8Bs/7qcjz2bN9q71ubV5D3RpW1ZuLl6tes1Yl3gEMAADCLDyKq2tNC2IYhi5mXNa5tKwrv9Iz//z6z9/Ts7Q/8UKR7anOp2fpzbUH873Pz8ekiNBAVatozhVMA1W94l9fR4SaFRbkb9eHqNrXraoaYYFOrWPNj6+PSQ3DDPW5rob8/fPfjQsAAHsRZuExnFlrmsMwDKVmZv8VQv8MpmfTsnT+z2Nn07J0Pj3n60ydT78y7rLFzkunRbihflV1qFfVegU14s8rrJWDA+RTisGdOlYAQHlAmIVHcLTW1DAMpWdl62zuq6P5XDHNCaZn/7z/fHqmXXWkBTH7+ahycIAqBfsrLMjf+nWl4ACdT8/UfzYfK/IcsTc3LLNP5dOjFQDg6QizTuSub4m7u6xsi9IuZ+lSVrbSMrN14VJWkS2kxi3arutqHdb5S38G1LSsIvugFibA10eVgq+E0bBgf1W6KphW+vNYztc59wX6+xZ4zmyLofX7ksr8bf2i0KMVAODJCLNOUhZvibtKtuXKVc+0zMtKz7wSONOzsq1fp2VetgbRtMzsq8ZczjM+97lSM3yVvXG1w3PKuGzR/34/m+e4v6/pSuAMuiqYVgiwvXJ6VTAN9PcpVvP+wrjz2/r0aAUAeCrCrBMUt/1SabFYDF26/FeQTM/6K2Tmvp1vyLw6nGZdOX4pM1tpfz4u83Lxr3gW7a8g52OSggP8ZDJJF+zoizqyc7R6NI24ElIrXAmwwQG+pR5KS4K39QEAKF2E2VJWVPslk6SpX+7WjQ2rKeOyxSZI5rlimXm5gOB5JZgWdIXzUpYzw+ZfTCYpyN9XwQG+Cvzz96AAPwXnHAvwtX4dFOBnHRsU8Ofv/jlf+yk4wFd+JkObfvhWt8X0VGgFswJ8r1wd3fjbGQ2et6nI+fRqFukRVxd5Wx8AgNJDmC1lmw8nF9p+yZB08vwlNYtfUSbzCfT3UfCfQTJviPRVkL+fNWDmhM2/wmne+3LCZ5C/b6m/FZ+VlaV9ZqlSsL/8/f6qPW1ft4pLWkg5E2/rAwBQOtwizM6ePVsvv/yyEhIS1LJlS82aNUvt27cvcPzHH3+sSZMm6ciRI2rYsKFefPFF9enTpwxnXLBTFwrvI3q1AD+fAq5YFnaF88+AGuB7Vdi0DZ9B/r6l2srJVdy51hQAALiWy8Ps4sWLFRcXp7lz56pDhw6aOXOmYmJitG/fPlWvXj3P+B9//FGDBw/W9OnT9be//U0LFy5U//79tXXrVjVv3twF34Ete3dLmj+8nW5qVM3uve29HbWmAAAgPy4PszNmzNCoUaM0cuRISdLcuXP11Vdfaf78+ZowYUKe8a+//rpuvfVW/eMf/5AkPffcc1q1apXefPNNzZ07t0znnh973xLv2rg6VxIdRK0pAAC4mkvDbGZmprZs2aKJEydaj/n4+KhHjx7auHFjvo/ZuHGj4uLibI7FxMTo888/z3d8RkaGMjIyrLdTUlIkXanPzMrKKuF3kL+nezfWmEU7CnxL/OnejWXJvixLtlOe3mPlrEdR69LumlBJoZLEz9ED2bvO8Hystfdgrb1DWa6zI8/h0jB7+vRpZWdnKyIiwuZ4RESE9u7dm+9jEhIS8h2fkJCQ7/jp06dr6tSpeY6vXLlSwcHBxZx50UY2MumzIz46l/nXVcOwAEN3RluU/fsWff27057a461atcrVU0AZYJ29B2vtPVhr71AW65yWlmb3WJeXGTjbxIkTba7kpqSkKCoqSr169VJoaKjTnrePpPEWQz//flanLmSoekWz2tWpzFvihcjKytKqVavUs2dP+fv7u3o6cBLW2Xuw1t6DtfYOZbnOOe+k28OlYTY8PFy+vr5KTEy0OZ6YmKjIyMh8HxMZGenQeLPZLLPZnOe4v7+/0xfCX1KXRhFFjoOtslgbuB7r7D1Ya+/BWnuHMslQDpzfpR+lDwgIUNu2bbVmzRrrMYvFojVr1qhTp075PqZTp04246Url7sLGg8AAIDyy+VlBnFxcRo+fLjatWun9u3ba+bMmUpNTbV2Nxg2bJhq1aql6dOnS5LGjRunrl276tVXX9Vtt92mRYsW6eeff9Y777zjym8DAAAALuDyMDto0CAlJSVp8uTJSkhIUKtWrbR8+XLrh7yOHj0qH5+/LiB37txZCxcu1DPPPKOnnnpKDRs21Oeff+4WPWYBAABQtlweZiUpNjZWsbGx+d63fv36PMcGDBigAQMGOHlWAAAAcHdsPwUAAACPRZgFAACAxyLMAgAAwGMRZgEAAOCxCLMAAADwWIRZAAAAeCy3aM1VlgzDkOTYnr8oG1lZWUpLS1NKSgrbIZZjrLP3YK29B2vtHcpynXNyWk5uK4zXhdkLFy5IkqKiolw8EwAAABTmwoULCgsLK3SMybAn8pYjFotFJ06cUMWKFWUymVw9HeSSkpKiqKgoHTt2TKGhoa6eDpyEdfYerLX3YK29Q1mus2EYunDhgmrWrGmzE2x+vO7KrI+Pj2rXru3qaaAQoaGh/GXoBVhn78Faew/W2juU1ToXdUU2Bx8AAwAAgMcizAIAAMBjEWbhNsxms+Lj42U2m109FTgR6+w9WGvvwVp7B3ddZ6/7ABgAAADKD67MAgAAwGMRZgEAAOCxCLMAAADwWIRZAAAAeCzCLNxaRkaGWrVqJZPJpO3bt7t6OnCCfv366ZprrlFgYKBq1Kih++67TydOnHD1tFCKjhw5ogceeEB169ZVUFCQ6tevr/j4eGVmZrp6anCCadOmqXPnzgoODlalSpVcPR2UotmzZys6OlqBgYHq0KGDNm/e7OopSSLMws2NHz9eNWvWdPU04ETdu3fXkiVLtG/fPn366af67bffdPfdd7t6WihFe/fulcVi0dtvv61du3bptdde09y5c/XUU0+5empwgszMTA0YMECPPvqoq6eCUrR48WLFxcUpPj5eW7duVcuWLRUTE6NTp065emq05oL7+uabbxQXF6dPP/1UzZo107Zt29SqVStXTwtOtmzZMvXv318ZGRny9/d39XTgJC+//LLmzJmjQ4cOuXoqcJL3339ff//733Xu3DlXTwWloEOHDrr++uv15ptvSpIsFouioqI0ZswYTZgwwaVz48os3FJiYqJGjRqlf//73woODnb1dFBGkpOT9dFHH6lz584E2XLu/PnzqlKliqunAcAOmZmZ2rJli3r06GE95uPjox49emjjxo0unNmfc3H1BICrGYahESNG6JFHHlG7du1cPR2UgSeffFIVKlRQ1apVdfToUX3xxReunhKc6ODBg5o1a5YefvhhV08FgB1Onz6t7OxsRURE2ByPiIhQQkKCi2b1F8IsysyECRNkMpkK/bV3717NmjVLFy5c0MSJE109ZRSTvWud4x//+Ie2bdumlStXytfXV8OGDRMVUO7P0XWWpOPHj+vWW2/VgAEDNGrUKBfNHI4qzloDZYWaWZSZpKQknTlzptAx9erV08CBA/Xll1/KZDJZj2dnZ8vX11dDhw7VBx984OypooTsXeuAgIA8x//44w9FRUXpxx9/VKdOnZw1RZQCR9f5xIkT6tatmzp27Kj3339fPj5cT/EUxfkzTc1s+ZGZmang4GB98skn6t+/v/X48OHDde7cOZe/m+bn0meHV6lWrZqqVatW5Lg33nhDzz//vPX2iRMnFBMTo8WLF6tDhw7OnCJKib1rnR+LxSLpSls2uDdH1vn48ePq3r272rZtqwULFhBkPUxJ/kzD8wUEBKht27Zas2aNNcxaLBatWbNGsbGxrp2cCLNwQ9dcc43N7ZCQEElS/fr1Vbt2bVdMCU7y008/6X//+5+6dOmiypUr67ffftOkSZNUv359rsqWI8ePH1e3bt1Up04dvfLKK0pKSrLeFxkZ6cKZwRmOHj2q5ORkHT16VNnZ2dYe4Q0aNLD+fQ7PExcXp+HDh6tdu3Zq3769Zs6cqdTUVI0cOdLVUyPMAnCd4OBgffbZZ4qPj1dqaqpq1KihW2+9Vc8884zMZrOrp4dSsmrVKh08eFAHDx7M8x9SKt3Kn8mTJ9uUg7Vu3VqStG7dOnXr1s1Fs0JJDRo0SElJSZo8ebISEhLUqlUrLV++PM+HwlyBmlkAAAB4LIqWAAAA4LEIswAAAPBYhFkAAAB4LMIsAAAAPBZhFgAAAB6LMAsAAACPRZgFAACAxyLMAgAAwGMRZgHASd5//31VqlSpxOeJjo7WzJkzCx1jMpn0+eefS5KOHDkik8lk3UZ0/fr1MplMOnfuXInnkp9JkybpoYcest7u1q2b/v73vzvluW666SYtXLjQ7vGnT59W9erV9ccff9gcv+eee/Tqq6+W9vQAuABhFoBbGTFihPr371+mz/n+++/LZDLJZDLJx8dHtWvX1siRI3Xq1KkynUdJnDx5Ur179873vs6dO+vkyZMKCwuTVHohW5ISEhL0+uuv6+mnny6V8xVm2bJlSkxM1D333GP3Y8LDwzVs2DDFx8fbHH/mmWc0bdo0nT9/vrSnCaCMEWYBQFJoaKhOnjypP/74Q/PmzdM333yj++67L9+x2dnZslgsZTzDwkVGRspsNud7X0BAgCIjI2UymUr9ed9991117txZderUKfVzX+2NN97QyJEj5ePj2D9dI0eO1EcffaTk5GTrsebNm6t+/fr68MMPS3uaAMoYYRaAR/n222/Vvn17mc1m1ahRQxMmTNDly5et91+4cEFDhw5VhQoVVKNGDb322mt2ve1tMpkUGRmpmjVrqnfv3ho7dqxWr16t9PR065XMZcuWqWnTpjKbzTp69KjOnj2rYcOGqXLlygoODlbv3r114MCBPOf+/PPP1bBhQwUGBiomJkbHjh2z3vfbb7/p9ttvV0REhEJCQnT99ddr9erVec5x4cIFDR48WBUqVFCtWrU0e/bsPPPPKTO4Wu4yg/Xr12vkyJE6f/689Wr0lClT9Oyzz6p58+Z5HtuqVStNmjSpwJ/bokWL1Ldv3wLvl2TXz2nevHmKiopScHCw7rjjDs2YMcPm6nFSUpLWrl2b57lMJpPmzJmj3r17KygoSPXq1dMnn3xiM6ZZs2aqWbOmli5danO8b9++WrRoUaFzB+D+CLMAPMbx48fVp08fXX/99dqxY4fmzJmj9957T88//7x1TFxcnDZs2KBly5Zp1apV+v7777V161aHnysoKEgWi8UalNPS0vTiiy/q3Xff1a5du1S9enWNGDFCP//8s5YtW6aNGzfKMAz16dNHWVlZ1vOkpaVp2rRp+te//qUNGzbo3LlzNm+TX7x4UX369NGaNWu0bds23Xrrrerbt6+OHj1qM5+XX35ZLVu21LZt2zRhwgSNGzdOq1atcvj76ty5s2bOnGm9En3y5Ek98cQTuv/++7Vnzx7973//s47dtm2bfvnlF40cOTLfcyUnJ2v37t1q165doc9Z1M9pw4YNeuSRRzRu3Dht375dPXv21LRp02zO8cMPPyg4OFhNmjTJc/5Jkybprrvu0o4dOzR06FDdc8892rNnj82Y9u3b6/vvv89zbPPmzcrIyCh0/gDcnAEAbmT48OHG7bffnu99Tz31lNG4cWPDYrFYj82ePdsICQkxsrOzjZSUFMPf39/4+OOPrfefO3fOCA4ONsaNG1fgcy5YsMAICwuz3t6/f7/RqFEjo127dtb7JRnbt2+3GSPJ2LBhg/XY6dOnjaCgIGPJkiU2j9u0aZN1zJ49ewxJxk8//VTgfJo1a2bMmjXLertOnTrGrbfeajNm0KBBRu/eva23JRlLly41DMMwDh8+bEgytm3bZhiGYaxbt86QZJw9ezbf7zdH7969jUcffdR6e8yYMUa3bt0KnOe2bdsMScbRo0dtjnft2tX687bn5zRo0CDjtttusznH0KFDbeb42muvGfXq1cszB0nGI488YnOsQ4cONt+HYRjG448/nud72bFjhyHJOHLkSIHfIwD3x5VZAB5jz5496tSpk03t5w033KCLFy/qjz/+0KFDh5SVlaX27dtb7w8LC1Pjxo2LPPf58+cVEhKi4OBgNW7cWBEREfroo4+s9wcEBOi6666zmYufn586dOhgPVa1alU1btzY5qqgn5+frr/+euvta6+9VpUqVbKOuXjxop544gk1adJElSpVUkhIiPbs2ZPnymynTp3y3L766mNJjRo1Sv/5z3906dIlZWZmauHChbr//vsLHJ+eni5JCgwMLHCMPT+nffv22ayZpDy309PTC3wee342QUFBSktLy3NMUp7jADyLn6snAADuoGLFitq6dat8fHxUo0YNa9DJERQU5JQPUD3xxBNatWqVXnnlFTVo0EBBQUG6++67lZmZWerPVZS+ffvKbDZr6dKlCggIUFZWlu6+++4Cx4eHh0u6UhNbrVo1p84tPDxcZ8+eLfbjk5OT88wx5wNhzp47AOfiyiwAj9GkSRNrzWWODRs2qGLFiqpdu7bq1asnf39/m7rP8+fPa//+/UWe28fHRw0aNFC9evXyBNmC5nL58mX99NNP1mNnzpzRvn371LRpU+uxy5cv6+eff7be3rdvn86dO2et/dywYYNGjBihO+64Qy1atFBkZKSOHDmS5/k2bdqU53Z+9aP2CAgIUHZ2dp7jfn5+Gj58uBYsWKAFCxbonnvuKfRnUb9+fYWGhmr37t0FjrHn59S4cWObNZOU53br1q2VkJCQb6C152ezc+dOtW7dOs+x2rVrW0M5AM9EmAXgds6fP6/t27fb/Dp27Jgee+wxHTt2TGPGjNHevXv1xRdfKD4+XnFxcfLx8VHFihU1fPhw/eMf/9C6deu0a9cuPfDAA/Lx8Sn1q6oNGzbU7bffrlGjRumHH37Qjh07dO+996pWrVq6/fbbreP8/f01ZswY/fTTT9qyZYtGjBihjh07Wt9Gb9iwoT777DNt375dO3bs0JAhQ/Jt+7Vhwwa99NJL2r9/v2bPnq2PP/5Y48aNK9bco6OjdfHiRa1Zs0anT5+2eZv9wQcf1Nq1a7V8+fJCSwykK/8B6NGjh3744YcCx9jzcxozZoy+/vprzZgxQwcOHNDbb7+tb775xmbNWrdurfDwcG3YsCHPc3z88ceaP3++9u/fr/j4eG3evFmxsbHW+9PS0rRlyxb16tXL5nHff/99nmMAPJCri3YBILfhw4cbkvL8euCBBwzDMIz169cb119/vREQEGBERkYaTz75pJGVlWV9fEpKijFkyBAjODjYiIyMNGbMmGG0b9/emDBhQoHPWdAHooq6Pzk52bjvvvuMsLAwIygoyIiJiTH279+f53GffvqpUa9ePcNsNhs9evQwfv/9d+uYw4cPG927dzeCgoKMqKgo480337T5AJVhXPkA2NSpU40BAwZYv6/XX3/dZi5y4ANghmEYjzzyiFG1alVDkhEfH29zrhtvvNFo1qxZgT+P3L7++mujVq1aRnZ2tvXY1fMv6udkGIbxzjvvGLVq1TKCgoKM/v37G88//7wRGRlpM2b8+PHGPffck+f7nj17ttGzZ0/DbDYb0dHRxuLFi23GLFy40GjcuLHNsfT0dCMsLMzYuHGjXd8nAPdlMoxc79cBQDmTmpqqWrVq6dVXX9UDDzzg6um4PcMw1LBhQz322GOKi4uza3yHDh30+OOPa/DgwaU2j1GjRmnv3r027bQSEhLUrFkzbd261bpJg8lk0tKlSwvdNa5jx44aO3ashgwZYj02Z84cLV26VCtXriy1OQNwDT4ABqBc2bZtm/bu3av27dvr/PnzevbZZyXJ5q1/5C8pKUmLFi1SQkJCgb1lr2YymfTOO+/o119/LdFzv/LKK+rZs6cqVKigb775Rh988IHeeustmzGRkZF67733dPToUbt3HDt9+rTuvPPOPEHb399fs2bNKtGcAbgHrswCKFe2bdumBx98UPv27VNAQIDatm2rGTNmqEWLFq6emtszmUwKDw/X66+/bnMVsywMHDhQ69ev14ULF1SvXj2NGTNGjzzySJGPs+fKLIDyjTALAAAAj0U3AwAAAHgswiwAAAA8FmEWAAAAHoswCwAAAI9FmAUAAIDHIswCAADAYxFmAQAA4LEIswAAAPBY/w9cYhl5FMu3jAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 800x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Example probabilities (ensure no zero values to avoid log issues)\n",
    "p = np.arange(0.01, 1, 0.01)\n",
    "logp = np.log(p)\n",
    "\n",
    "# Plot: x-axis is log(p), y-axis is p\n",
    "plt.figure(figsize=(8, 5))\n",
    "plt.plot(logp, p, marker='o')\n",
    "plt.title('Log Probabilities vs Probabilities')\n",
    "plt.xlabel('Log Probability (log(p))')\n",
    "plt.ylabel('Probability (p)')\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "20e991c7-44bf-4ae2-8a7c-1480aea33aaf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:11:30.641968Z",
     "iopub.status.busy": "2024-12-06T12:11:30.641744Z",
     "iopub.status.idle": "2024-12-06T12:11:30.645549Z",
     "shell.execute_reply": "2024-12-06T12:11:30.644746Z",
     "shell.execute_reply.started": "2024-12-06T12:11:30.641953Z"
    }
   },
   "outputs": [],
   "source": [
    "# !pip install -U openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d6ac6411-d4da-47ce-9259-57e878ab5edb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:54:10.516598Z",
     "iopub.status.busy": "2024-12-07T11:54:10.515945Z",
     "iopub.status.idle": "2024-12-07T11:54:11.180998Z",
     "shell.execute_reply": "2024-12-07T11:54:11.179293Z",
     "shell.execute_reply.started": "2024-12-07T11:54:10.516551Z"
    }
   },
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from math import exp\n",
    "import numpy as np\n",
    "from IPython.display import display, HTML\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4b26d4d5-ed60-4f01-be63-5ac5f100a87c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:54:12.454400Z",
     "iopub.status.busy": "2024-12-07T11:54:12.454018Z",
     "iopub.status.idle": "2024-12-07T11:54:12.475957Z",
     "shell.execute_reply": "2024-12-07T11:54:12.473686Z",
     "shell.execute_reply.started": "2024-12-07T11:54:12.454377Z"
    }
   },
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "assert load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "634ecc03-d22a-434a-8419-ae91a8a76014",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:54:13.791656Z",
     "iopub.status.busy": "2024-12-07T11:54:13.791040Z",
     "iopub.status.idle": "2024-12-07T11:54:13.894962Z",
     "shell.execute_reply": "2024-12-07T11:54:13.893272Z",
     "shell.execute_reply.started": "2024-12-07T11:54:13.791611Z"
    }
   },
   "outputs": [],
   "source": [
    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5dc09348-386f-46b3-ba08-fc959f310cef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:54:15.265919Z",
     "iopub.status.busy": "2024-12-07T11:54:15.264628Z",
     "iopub.status.idle": "2024-12-07T11:54:15.275224Z",
     "shell.execute_reply": "2024-12-07T11:54:15.273180Z",
     "shell.execute_reply.started": "2024-12-07T11:54:15.265883Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_completion(\n",
    "    messages: list[dict[str, str]],\n",
    "    model: str = \"gpt-4o-2024-08-06\",\n",
    "    max_tokens=500,\n",
    "    temperature=0,\n",
    "    stop=None,\n",
    "    seed=123,\n",
    "    tools=None,\n",
    "    logprobs=None,  # whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message..\n",
    "    top_logprobs=None,\n",
    ") -> str:\n",
    "    params = {\n",
    "        \"model\": model,\n",
    "        \"messages\": messages,\n",
    "        \"max_tokens\": max_tokens,\n",
    "        \"temperature\": temperature,\n",
    "        \"stop\": stop,\n",
    "        \"seed\": seed,\n",
    "        \"logprobs\": logprobs,\n",
    "        \"top_logprobs\": top_logprobs,\n",
    "    }\n",
    "    if tools:\n",
    "        params[\"tools\"] = tools\n",
    "\n",
    "    completion = client.chat.completions.create(**params)\n",
    "    return completion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9918f6e7-7236-4f8b-9d9c-78fcd9eff2c3",
   "metadata": {},
   "source": [
    "### 1. Using logprobs to assess confidence for classification tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adba130e-c2d5-4884-a180-036350f6157f",
   "metadata": {},
   "source": [
    "- Let's say we want to create a system to classify news articles into a set of pre-defined categories. Without logprobs, we can use Chat Completions to do this, but it is much more difficult to **assess the certainty** with which the model made its classifications.\n",
    "- Now, with logprobs enabled, we can see exactly **how confident** the model is in its predictions, which is crucial for creating an accurate and trustworthy classifier.\n",
    "    - For example, if the log probability for the chosen category is high, this suggests the model is quite confident in its classification.\n",
    "    - If it's low, this suggests the model is less confident.\n",
    "    - This can be particularly useful in cases where the model's classification is not what you expected, or when the model's output needs to be reviewed or validated by a human."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5e89f9cf-9ce5-4712-8584-3f93b872a271",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:54:48.485770Z",
     "iopub.status.busy": "2024-12-07T11:54:48.485140Z",
     "iopub.status.idle": "2024-12-07T11:54:48.494757Z",
     "shell.execute_reply": "2024-12-07T11:54:48.492574Z",
     "shell.execute_reply.started": "2024-12-07T11:54:48.485722Z"
    }
   },
   "outputs": [],
   "source": [
    "CLASSIFICATION_PROMPT = \"\"\"You will be given a headline of a news article.\n",
    "Classify the article into one of the following categories: Technology, Politics, Sports, and Art.\n",
    "Return only the name of the category, and nothing else.\n",
    "MAKE SURE your output is one of the four categories stated.\n",
    "Article headline: {headline}\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8a18cadd-daed-479c-a73d-7799c99ccf05",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:54:47.345822Z",
     "iopub.status.busy": "2024-12-07T11:54:47.345506Z",
     "iopub.status.idle": "2024-12-07T11:54:47.353640Z",
     "shell.execute_reply": "2024-12-07T11:54:47.351694Z",
     "shell.execute_reply.started": "2024-12-07T11:54:47.345800Z"
    }
   },
   "outputs": [],
   "source": [
    "headlines = [\n",
    "    \"Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.\",\n",
    "    \"Local Mayor Launches Initiative to Enhance Urban Public Transport.\",\n",
    "    \"Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "21178c05-0abf-4899-9fbb-3eef2eb8a283",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:17:45.941849Z",
     "iopub.status.busy": "2024-12-06T12:17:45.941214Z",
     "iopub.status.idle": "2024-12-06T12:17:48.654846Z",
     "shell.execute_reply": "2024-12-06T12:17:48.652691Z",
     "shell.execute_reply.started": "2024-12-06T12:17:45.941802Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Headline: Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.\n",
      "Category: Technology\n",
      "\n",
      "\n",
      "Headline: Local Mayor Launches Initiative to Enhance Urban Public Transport.\n",
      "Category: Politics\n",
      "\n",
      "\n",
      "Headline: Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut\n",
      "Category: Art\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for headline in headlines:\n",
    "    print(f\"\\nHeadline: {headline}\")\n",
    "    API_RESPONSE = get_completion(\n",
    "        [{\"role\": \"user\", \"content\": CLASSIFICATION_PROMPT.format(headline=headline)}],\n",
    "    )\n",
    "    print(f\"Category: {API_RESPONSE.choices[0].message.content}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "274efb14-69d0-4d98-bc3b-159b52ce47a9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:32:00.646998Z",
     "iopub.status.busy": "2024-12-06T12:32:00.646668Z",
     "iopub.status.idle": "2024-12-06T12:32:01.627025Z",
     "shell.execute_reply": "2024-12-06T12:32:01.624890Z",
     "shell.execute_reply.started": "2024-12-06T12:32:00.646972Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletion</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chatcmpl-AbRmzsYyvSmgWYf8udpB4fdnOxPyb'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">choices</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Choice</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">finish_reason</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'stop'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">index</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprobs</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChoiceLogprobs</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Technology'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">84</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">99</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Technology'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">84</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">99</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' Technology'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">84</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">99</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-19.375</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">message</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionMessage</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Technology'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">role</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'assistant'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">audio</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">function_call</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">tool_calls</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">created</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1733488321</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">model</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'gpt-4o-2024-08-06'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">object</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chat.completion'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">service_tier</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">system_fingerprint</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'fp_c7ca0ebaca'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">usage</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionUsage</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">81</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">total_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">82</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens_details</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionTokensDetails</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">accepted_prediction_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">audio_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">reasoning_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">rejected_prediction_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens_details</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">PromptTokensDetails</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">audio_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #808000; text-decoration-color: #808000\">cached_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mChatCompletion\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mid\u001b[0m=\u001b[32m'chatcmpl-AbRmzsYyvSmgWYf8udpB4fdnOxPyb'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mchoices\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;35mChoice\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mfinish_reason\u001b[0m=\u001b[32m'stop'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mindex\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mlogprobs\u001b[0m=\u001b[1;35mChoiceLogprobs\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'Technology'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m84\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m99\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m103\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'Technology'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m84\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m99\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m103\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.0\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' Technology'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m84\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m99\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m103\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-19.375\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mmessage\u001b[0m=\u001b[1;35mChatCompletionMessage\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[32m'Technology'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrole\u001b[0m=\u001b[32m'assistant'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33maudio\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mfunction_call\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mtool_calls\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mcreated\u001b[0m=\u001b[1;36m1733488321\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mmodel\u001b[0m=\u001b[32m'gpt-4o-2024-08-06'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mobject\u001b[0m=\u001b[32m'chat.completion'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mservice_tier\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33msystem_fingerprint\u001b[0m=\u001b[32m'fp_c7ca0ebaca'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33musage\u001b[0m=\u001b[1;35mCompletionUsage\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mcompletion_tokens\u001b[0m=\u001b[1;36m1\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mprompt_tokens\u001b[0m=\u001b[1;36m81\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtotal_tokens\u001b[0m=\u001b[1;36m82\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mcompletion_tokens_details\u001b[0m=\u001b[1;35mCompletionTokensDetails\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33maccepted_prediction_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33maudio_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mreasoning_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mrejected_prediction_tokens\u001b[0m=\u001b[1;36m0\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mprompt_tokens_details\u001b[0m=\u001b[1;35mPromptTokensDetails\u001b[0m\u001b[1m(\u001b[0m\u001b[33maudio_tokens\u001b[0m=\u001b[1;36m0\u001b[0m, \u001b[33mcached_tokens\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from rich.pretty import pprint\n",
    "pprint(\n",
    "get_completion(\n",
    "        [{\"role\": \"user\", \"content\": CLASSIFICATION_PROMPT.format(headline='Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.')}],\n",
    "        logprobs=True,\n",
    "        top_logprobs=2,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "f379ae27-1826-4494-80a6-696255c6f4c0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:07:19.171498Z",
     "iopub.status.busy": "2024-12-06T13:07:19.170835Z",
     "iopub.status.idle": "2024-12-06T13:07:19.185816Z",
     "shell.execute_reply": "2024-12-06T13:07:19.183872Z",
     "shell.execute_reply.started": "2024-12-06T13:07:19.171451Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Technology'"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bytes([84, 101, 99, 104, 110, 111, 108, 111, 103, 121]).decode(\"utf-8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "90660534-f726-4957-a85f-85ecef0aec71",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:26:12.044674Z",
     "iopub.status.busy": "2024-12-06T12:26:12.044007Z",
     "iopub.status.idle": "2024-12-06T12:26:14.203584Z",
     "shell.execute_reply": "2024-12-06T12:26:14.201442Z",
     "shell.execute_reply.started": "2024-12-06T12:26:12.044626Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Headline: Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<span style='color: green'>Generated response:</span> Technology<br><span style='color: cyan'>Output token 1:</span> 'Technology', <span style='color: darkorange'>logprobs:</span> 0.0, <span style='color: magenta'>linear probability:</span> 100.0%<br><span style='color: cyan'>Output token 2:</span> ' Technology', <span style='color: darkorange'>logprobs:</span> -19.25, <span style='color: magenta'>linear probability:</span> 0.0%<br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Headline: Local Mayor Launches Initiative to Enhance Urban Public Transport.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<span style='color: green'>Generated response:</span> Politics<br><span style='color: cyan'>Output token 1:</span> 'Politics', <span style='color: darkorange'>logprobs:</span> 0.0, <span style='color: magenta'>linear probability:</span> 100.0%<br><span style='color: cyan'>Output token 2:</span> 'Polit', <span style='color: darkorange'>logprobs:</span> -18.0, <span style='color: magenta'>linear probability:</span> 0.0%<br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Headline: Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<span style='color: green'>Generated response:</span> Art<br><span style='color: cyan'>Output token 1:</span> 'Art', <span style='color: darkorange'>logprobs:</span> -0.002037607, <span style='color: magenta'>linear probability:</span> 99.8%<br><span style='color: cyan'>Output token 2:</span> 'Sports', <span style='color: darkorange'>logprobs:</span> -6.3770375, <span style='color: magenta'>linear probability:</span> 0.17%<br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for headline in headlines:\n",
    "    print(f\"\\nHeadline: {headline}\")\n",
    "    API_RESPONSE = get_completion(\n",
    "        [{\"role\": \"user\", \"content\": CLASSIFICATION_PROMPT.format(headline=headline)}],\n",
    "        logprobs=True,\n",
    "        top_logprobs=2,\n",
    "    )\n",
    "    generated_response = API_RESPONSE.choices[0].message.content.replace('\\n', '<br>')\n",
    "    top_two_logprobs = API_RESPONSE.choices[0].logprobs.content[0].top_logprobs\n",
    "    \n",
    "    html_content = (\n",
    "        f\"<span style='color: green'>Generated response:</span> {generated_response}<br>\"\n",
    "    )\n",
    "    for i, logprob in enumerate(top_two_logprobs, start=1):\n",
    "        html_content += (\n",
    "            f\"<span style='color: cyan'>Output token {i}:</span> '{logprob.token}', \"\n",
    "            f\"<span style='color: darkorange'>logprobs:</span> {logprob.logprob}, \"\n",
    "            f\"<span style='color: magenta'>linear probability:</span> {np.round(np.exp(logprob.logprob)*100,2)}%<br>\"\n",
    "        )\n",
    "    display(HTML(html_content))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7994581b-ac39-4d76-8d02-2b156e47efa5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:53:53.344824Z",
     "iopub.status.busy": "2024-12-07T11:53:53.344405Z",
     "iopub.status.idle": "2024-12-07T11:53:53.355030Z",
     "shell.execute_reply": "2024-12-07T11:53:53.352994Z",
     "shell.execute_reply.started": "2024-12-07T11:53:53.344799Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.363462252943701e-09"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(-19.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ef2b416c-02b5-4127-a5b7-1c808f2e9da6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-07T11:54:53.372687Z",
     "iopub.status.busy": "2024-12-07T11:54:53.372042Z",
     "iopub.status.idle": "2024-12-07T11:54:55.250024Z",
     "shell.execute_reply": "2024-12-07T11:54:55.247904Z",
     "shell.execute_reply.started": "2024-12-07T11:54:53.372626Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Headline: Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<span style='color: green'>Generated response:</span> Technology<br><span style='color: cyan'>Output token 1:</span> 'Technology', <span style='color: darkorange'>logprobs:</span> 0.0, <span style='color: magenta'>linear probability:</span> 100.0%<br><span style='color: cyan'>Output token 2:</span> ' Technology', <span style='color: darkorange'>logprobs:</span> -19.0, <span style='color: magenta'>linear probability:</span> 0.0%<br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Headline: Local Mayor Launches Initiative to Enhance Urban Public Transport.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<span style='color: green'>Generated response:</span> Politics<br><span style='color: cyan'>Output token 1:</span> 'Politics', <span style='color: darkorange'>logprobs:</span> -3.1281633e-07, <span style='color: magenta'>linear probability:</span> 100.0%<br><span style='color: cyan'>Output token 2:</span> 'Polit', <span style='color: darkorange'>logprobs:</span> -16.0, <span style='color: magenta'>linear probability:</span> 0.0%<br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Headline: Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<span style='color: green'>Generated response:</span> Art<br><span style='color: cyan'>Output token 1:</span> 'Art', <span style='color: darkorange'>logprobs:</span> -0.032066498, <span style='color: magenta'>linear probability:</span> 96.84%<br><span style='color: cyan'>Output token 2:</span> 'Sports', <span style='color: darkorange'>logprobs:</span> -4.0320663, <span style='color: magenta'>linear probability:</span> 1.77%<br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for headline in headlines:\n",
    "    print(f\"\\nHeadline: {headline}\")\n",
    "    API_RESPONSE = get_completion(\n",
    "        [{\"role\": \"user\", \"content\": CLASSIFICATION_PROMPT.format(headline=headline)}],\n",
    "        model='gpt-4o-mini',\n",
    "        logprobs=True,\n",
    "        top_logprobs=2,\n",
    "    )\n",
    "    generated_response = API_RESPONSE.choices[0].message.content.replace('\\n', '<br>')\n",
    "    top_two_logprobs = API_RESPONSE.choices[0].logprobs.content[0].top_logprobs\n",
    "    \n",
    "    html_content = (\n",
    "        f\"<span style='color: green'>Generated response:</span> {generated_response}<br>\"\n",
    "    )\n",
    "    for i, logprob in enumerate(top_two_logprobs, start=1):\n",
    "        html_content += (\n",
    "            f\"<span style='color: cyan'>Output token {i}:</span> '{logprob.token}', \"\n",
    "            f\"<span style='color: darkorange'>logprobs:</span> {logprob.logprob}, \"\n",
    "            f\"<span style='color: magenta'>linear probability:</span> {np.round(np.exp(logprob.logprob)*100,2)}%<br>\"\n",
    "        )\n",
    "    display(HTML(html_content))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f784e76c-2d4e-45ce-86eb-0abd90bc1806",
   "metadata": {},
   "source": [
    "### 2. Retrieval confidence scoring to reduce hallucinations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a909ff69-9ac0-449f-a984-2fedf64517d8",
   "metadata": {},
   "source": [
    "- To reduce hallucinations, and the performance of our RAG-based Q&A system, we can use logprobs to evaluate how confident the model is in its retrieval.\n",
    "- `has_sufficient_context_for_answer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "bfba777e-ef03-4bdc-961a-ed41424d7f5c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:37:23.733532Z",
     "iopub.status.busy": "2024-12-06T12:37:23.732864Z",
     "iopub.status.idle": "2024-12-06T12:37:23.746761Z",
     "shell.execute_reply": "2024-12-06T12:37:23.744576Z",
     "shell.execute_reply.started": "2024-12-06T12:37:23.733485Z"
    }
   },
   "outputs": [],
   "source": [
    "# Article retrieved\n",
    "ada_lovelace_article = \"\"\"Augusta Ada King, Countess of Lovelace (née Byron; 10 December 1815 – 27 November 1852) was an English mathematician and writer, chiefly known for her work on Charles Babbage's proposed mechanical general-purpose computer, the Analytical Engine. She was the first to recognise that the machine had applications beyond pure calculation.\n",
    "Ada Byron was the only legitimate child of poet Lord Byron and reformer Lady Byron. All Lovelace's half-siblings, Lord Byron's other children, were born out of wedlock to other women. Byron separated from his wife a month after Ada was born and left England forever. He died in Greece when Ada was eight. Her mother was anxious about her upbringing and promoted Ada's interest in mathematics and logic in an effort to prevent her from developing her father's perceived insanity. Despite this, Ada remained interested in him, naming her two sons Byron and Gordon. Upon her death, she was buried next to him at her request. Although often ill in her childhood, Ada pursued her studies assiduously. She married William King in 1835. King was made Earl of Lovelace in 1838, Ada thereby becoming Countess of Lovelace.\n",
    "Her educational and social exploits brought her into contact with scientists such as Andrew Crosse, Charles Babbage, Sir David Brewster, Charles Wheatstone, Michael Faraday, and the author Charles Dickens, contacts which she used to further her education. Ada described her approach as \"poetical science\" and herself as an \"Analyst (& Metaphysician)\".\n",
    "When she was eighteen, her mathematical talents led her to a long working relationship and friendship with fellow British mathematician Charles Babbage, who is known as \"the father of computers\". She was in particular interested in Babbage's work on the Analytical Engine. Lovelace first met him in June 1833, through their mutual friend, and her private tutor, Mary Somerville.\n",
    "Between 1842 and 1843, Ada translated an article by the military engineer Luigi Menabrea (later Prime Minister of Italy) about the Analytical Engine, supplementing it with an elaborate set of seven notes, simply called \"Notes\".\n",
    "Lovelace's notes are important in the early history of computers, especially since the seventh one contained what many consider to be the first computer program—that is, an algorithm designed to be carried out by a machine. Other historians reject this perspective and point out that Babbage's personal notes from the years 1836/1837 contain the first programs for the engine. She also developed a vision of the capability of computers to go beyond mere calculating or number-crunching, while many others, including Babbage himself, focused only on those capabilities. Her mindset of \"poetical science\" led her to ask questions about the Analytical Engine (as shown in her notes) examining how individuals and society relate to technology as a collaborative tool.\n",
    "\"\"\"\n",
    "\n",
    "# Questions that can be easily answered given the article\n",
    "easy_questions = [\n",
    "    \"What nationality was Ada Lovelace?\",\n",
    "    \"What was an important finding from Lovelace's seventh note?\",\n",
    "]\n",
    "\n",
    "# Questions that are not fully covered in the article\n",
    "medium_questions = [\n",
    "    \"Did Lovelace collaborate with Charles Dickens\",\n",
    "    \"What concepts did Lovelace build with Charles Babbage\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6c744cd9-a183-438a-9843-b06e86a488ad",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:37:47.640578Z",
     "iopub.status.busy": "2024-12-06T12:37:47.639921Z",
     "iopub.status.idle": "2024-12-06T12:37:47.649965Z",
     "shell.execute_reply": "2024-12-06T12:37:47.647749Z",
     "shell.execute_reply.started": "2024-12-06T12:37:47.640530Z"
    }
   },
   "outputs": [],
   "source": [
    "# JUST\n",
    "PROMPT = \"\"\"You retrieved this article: {article}. The question is: {question}.\n",
    "Before even answering the question, consider whether you have sufficient information in the article to answer the question fully.\n",
    "Your output should JUST be the boolean true or false, of if you have sufficient information in the article to answer the question.\n",
    "Respond with just one word, the boolean true or false. You must output the word 'True', or the word 'False', nothing else.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "31657e3c-e1c9-4d26-a9c1-760cbdd62b9c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:39:12.915821Z",
     "iopub.status.busy": "2024-12-06T12:39:12.915169Z",
     "iopub.status.idle": "2024-12-06T12:39:13.880488Z",
     "shell.execute_reply": "2024-12-06T12:39:13.878175Z",
     "shell.execute_reply.started": "2024-12-06T12:39:12.915773Z"
    }
   },
   "outputs": [],
   "source": [
    "API_RESPONSE = get_completion(\n",
    "        [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": PROMPT.format(\n",
    "                    article=ada_lovelace_article, question=\"Did Lovelace collaborate with Charles Dickens\"\n",
    "                ),\n",
    "            }\n",
    "        ],\n",
    "        model=\"gpt-4\",\n",
    "        logprobs=True,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ba2125b3-98ff-40ee-8644-a04a76819c07",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:39:19.321129Z",
     "iopub.status.busy": "2024-12-06T12:39:19.320548Z",
     "iopub.status.idle": "2024-12-06T12:39:19.355204Z",
     "shell.execute_reply": "2024-12-06T12:39:19.353101Z",
     "shell.execute_reply.started": "2024-12-06T12:39:19.321083Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletion</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chatcmpl-AbRtxaI8UkXs9rrr7PCJBS3pAhkc3'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">choices</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Choice</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">finish_reason</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'stop'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">index</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprobs</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChoiceLogprobs</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'True'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">84</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.06760397</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">message</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionMessage</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'True'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">role</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'assistant'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">audio</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">function_call</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">tool_calls</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">created</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1733488753</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">model</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'gpt-4-0613'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">object</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chat.completion'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">service_tier</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">system_fingerprint</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">usage</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionUsage</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">698</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">total_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">699</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens_details</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionTokensDetails</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">accepted_prediction_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">audio_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">reasoning_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">rejected_prediction_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens_details</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">PromptTokensDetails</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">audio_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #808000; text-decoration-color: #808000\">cached_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mChatCompletion\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mid\u001b[0m=\u001b[32m'chatcmpl-AbRtxaI8UkXs9rrr7PCJBS3pAhkc3'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mchoices\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;35mChoice\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mfinish_reason\u001b[0m=\u001b[32m'stop'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mindex\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mlogprobs\u001b[0m=\u001b[1;35mChoiceLogprobs\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'True'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m84\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m117\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.06760397\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mmessage\u001b[0m=\u001b[1;35mChatCompletionMessage\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[32m'True'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrole\u001b[0m=\u001b[32m'assistant'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33maudio\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mfunction_call\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mtool_calls\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mcreated\u001b[0m=\u001b[1;36m1733488753\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mmodel\u001b[0m=\u001b[32m'gpt-4-0613'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mobject\u001b[0m=\u001b[32m'chat.completion'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mservice_tier\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33msystem_fingerprint\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33musage\u001b[0m=\u001b[1;35mCompletionUsage\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mcompletion_tokens\u001b[0m=\u001b[1;36m1\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mprompt_tokens\u001b[0m=\u001b[1;36m698\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtotal_tokens\u001b[0m=\u001b[1;36m699\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mcompletion_tokens_details\u001b[0m=\u001b[1;35mCompletionTokensDetails\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33maccepted_prediction_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33maudio_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mreasoning_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mrejected_prediction_tokens\u001b[0m=\u001b[1;36m0\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mprompt_tokens_details\u001b[0m=\u001b[1;35mPromptTokensDetails\u001b[0m\u001b[1m(\u001b[0m\u001b[33maudio_tokens\u001b[0m=\u001b[1;36m0\u001b[0m, \u001b[33mcached_tokens\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pprint(API_RESPONSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "ae31fc8e-f62e-4ac2-abd8-df939bf54f8c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:43:22.045086Z",
     "iopub.status.busy": "2024-12-06T12:43:22.044491Z",
     "iopub.status.idle": "2024-12-06T12:43:23.069426Z",
     "shell.execute_reply": "2024-12-06T12:43:23.067214Z",
     "shell.execute_reply.started": "2024-12-06T12:43:22.045038Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletion</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chatcmpl-AbRxyt8e62V0e5cMdR3y4b3DBUbMX'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">choices</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Choice</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">finish_reason</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'stop'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">index</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprobs</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChoiceLogprobs</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'True'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">84</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.51145124</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'True'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">84</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.51145124</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'False'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">70</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.9153552</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'false'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-14.718182</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">message</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionMessage</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'True'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">role</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'assistant'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">audio</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">function_call</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">tool_calls</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">created</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1733489002</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">model</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'gpt-4-0613'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">object</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chat.completion'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">service_tier</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">system_fingerprint</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">usage</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionUsage</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">701</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">total_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">702</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens_details</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionTokensDetails</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">accepted_prediction_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">audio_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">reasoning_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">rejected_prediction_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens_details</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">PromptTokensDetails</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">audio_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #808000; text-decoration-color: #808000\">cached_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mChatCompletion\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mid\u001b[0m=\u001b[32m'chatcmpl-AbRxyt8e62V0e5cMdR3y4b3DBUbMX'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mchoices\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;35mChoice\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mfinish_reason\u001b[0m=\u001b[32m'stop'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mindex\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mlogprobs\u001b[0m=\u001b[1;35mChoiceLogprobs\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'True'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m84\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m117\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.51145124\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'True'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m84\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m117\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.51145124\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'False'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m70\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.9153552\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'false'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m102\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-14.718182\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mmessage\u001b[0m=\u001b[1;35mChatCompletionMessage\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[32m'True'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrole\u001b[0m=\u001b[32m'assistant'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33maudio\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mfunction_call\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mtool_calls\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mcreated\u001b[0m=\u001b[1;36m1733489002\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mmodel\u001b[0m=\u001b[32m'gpt-4-0613'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mobject\u001b[0m=\u001b[32m'chat.completion'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mservice_tier\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33msystem_fingerprint\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33musage\u001b[0m=\u001b[1;35mCompletionUsage\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mcompletion_tokens\u001b[0m=\u001b[1;36m1\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mprompt_tokens\u001b[0m=\u001b[1;36m701\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtotal_tokens\u001b[0m=\u001b[1;36m702\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mcompletion_tokens_details\u001b[0m=\u001b[1;35mCompletionTokensDetails\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33maccepted_prediction_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33maudio_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mreasoning_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mrejected_prediction_tokens\u001b[0m=\u001b[1;36m0\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mprompt_tokens_details\u001b[0m=\u001b[1;35mPromptTokensDetails\u001b[0m\u001b[1m(\u001b[0m\u001b[33maudio_tokens\u001b[0m=\u001b[1;36m0\u001b[0m, \u001b[33mcached_tokens\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pprint(get_completion(\n",
    "        [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": PROMPT.format(\n",
    "                    article=ada_lovelace_article, question=question\n",
    "                ),\n",
    "            }\n",
    "        ],\n",
    "        model=\"gpt-4\",\n",
    "        logprobs=True,\n",
    "        top_logprobs=3,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8a46911f-39cf-48bd-a0ff-6a3e571d4c5f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:38:29.739046Z",
     "iopub.status.busy": "2024-12-06T12:38:29.738405Z",
     "iopub.status.idle": "2024-12-06T12:38:33.212750Z",
     "shell.execute_reply": "2024-12-06T12:38:33.210657Z",
     "shell.execute_reply.started": "2024-12-06T12:38:29.738998Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Questions clearly answered in article<p style=\"color:green\">Question: What nationality was Ada Lovelace?</p><p style=\"color:cyan\">has_sufficient_context_for_answer: True, <span style=\"color:darkorange\">logprobs: -1.9361265e-07, <span style=\"color:magenta\">linear probability: 100.0%</span></p><p style=\"color:green\">Question: What was an important finding from Lovelace's seventh note?</p><p style=\"color:cyan\">has_sufficient_context_for_answer: True, <span style=\"color:darkorange\">logprobs: -4.3202e-07, <span style=\"color:magenta\">linear probability: 100.0%</span></p>Questions only partially covered in the article<p style=\"color:green\">Question: Did Lovelace collaborate with Charles Dickens</p><p style=\"color:cyan\">has_sufficient_context_for_answer: True, <span style=\"color:darkorange\">logprobs: -0.027298978, <span style=\"color:magenta\">linear probability: 97.31%</span></p><p style=\"color:green\">Question: What concepts did Lovelace build with Charles Babbage</p><p style=\"color:cyan\">has_sufficient_context_for_answer: True, <span style=\"color:darkorange\">logprobs: -0.29815888, <span style=\"color:magenta\">linear probability: 74.22%</span></p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "html_output = \"\"\n",
    "html_output += \"Questions clearly answered in article\"\n",
    "\n",
    "for question in easy_questions:\n",
    "    API_RESPONSE = get_completion(\n",
    "        [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": PROMPT.format(\n",
    "                    article=ada_lovelace_article, question=question\n",
    "                ),\n",
    "            }\n",
    "        ],\n",
    "        model=\"gpt-4\",\n",
    "        logprobs=True,\n",
    "    )\n",
    "    html_output += f'<p style=\"color:green\">Question: {question}</p>'\n",
    "    for logprob in API_RESPONSE.choices[0].logprobs.content:\n",
    "        html_output += f'<p style=\"color:cyan\">has_sufficient_context_for_answer: {logprob.token}, <span style=\"color:darkorange\">logprobs: {logprob.logprob}, <span style=\"color:magenta\">linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%</span></p>'\n",
    "\n",
    "html_output += \"Questions only partially covered in the article\"\n",
    "\n",
    "for question in medium_questions:\n",
    "    API_RESPONSE = get_completion(\n",
    "        [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": PROMPT.format(\n",
    "                    article=ada_lovelace_article, question=question\n",
    "                ),\n",
    "            }\n",
    "        ],\n",
    "        model=\"gpt-4\",\n",
    "        logprobs=True,\n",
    "        top_logprobs=3,\n",
    "    )\n",
    "    html_output += f'<p style=\"color:green\">Question: {question}</p>'\n",
    "    for logprob in API_RESPONSE.choices[0].logprobs.content:\n",
    "        html_output += f'<p style=\"color:cyan\">has_sufficient_context_for_answer: {logprob.token}, <span style=\"color:darkorange\">logprobs: {logprob.logprob}, <span style=\"color:magenta\">linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%</span></p>'\n",
    "\n",
    "display(HTML(html_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "56807f5b-e2d9-4526-91c5-73e5df68533f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:42:09.744372Z",
     "iopub.status.busy": "2024-12-06T12:42:09.743707Z",
     "iopub.status.idle": "2024-12-06T12:42:12.330458Z",
     "shell.execute_reply": "2024-12-06T12:42:12.328310Z",
     "shell.execute_reply.started": "2024-12-06T12:42:09.744325Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Questions clearly answered in article<p style=\"color:green\">Question: What nationality was Ada Lovelace?</p><p style=\"color:cyan\">has_sufficient_context_for_answer: True, <span style=\"color:darkorange\">logprobs: -2.220075e-06, <span style=\"color:magenta\">linear probability: 100.0%</span></p><p style=\"color:green\">Question: What was an important finding from Lovelace's seventh note?</p><p style=\"color:cyan\">has_sufficient_context_for_answer: True, <span style=\"color:darkorange\">logprobs: -2.2484697e-05, <span style=\"color:magenta\">linear probability: 100.0%</span></p>Questions only partially covered in the article<p style=\"color:green\">Question: Did Lovelace collaborate with Charles Dickens</p><p style=\"color:cyan\">has_sufficient_context_for_answer: False, <span style=\"color:darkorange\">logprobs: -0.0011922525, <span style=\"color:magenta\">linear probability: 99.88%</span></p><p style=\"color:green\">Question: What concepts did Lovelace build with Charles Babbage</p><p style=\"color:cyan\">has_sufficient_context_for_answer: True, <span style=\"color:darkorange\">logprobs: -0.008621786, <span style=\"color:magenta\">linear probability: 99.14%</span></p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "html_output = \"\"\n",
    "html_output += \"Questions clearly answered in article\"\n",
    "\n",
    "for question in easy_questions:\n",
    "    API_RESPONSE = get_completion(\n",
    "        [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": PROMPT.format(\n",
    "                    article=ada_lovelace_article, question=question\n",
    "                ),\n",
    "            }\n",
    "        ],\n",
    "        logprobs=True,\n",
    "    )\n",
    "    html_output += f'<p style=\"color:green\">Question: {question}</p>'\n",
    "    for logprob in API_RESPONSE.choices[0].logprobs.content:\n",
    "        html_output += f'<p style=\"color:cyan\">has_sufficient_context_for_answer: {logprob.token}, <span style=\"color:darkorange\">logprobs: {logprob.logprob}, <span style=\"color:magenta\">linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%</span></p>'\n",
    "\n",
    "html_output += \"Questions only partially covered in the article\"\n",
    "\n",
    "for question in medium_questions:\n",
    "    API_RESPONSE = get_completion(\n",
    "        [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": PROMPT.format(\n",
    "                    article=ada_lovelace_article, question=question\n",
    "                ),\n",
    "            }\n",
    "        ],\n",
    "        logprobs=True,\n",
    "        top_logprobs=3,\n",
    "    )\n",
    "    html_output += f'<p style=\"color:green\">Question: {question}</p>'\n",
    "    for logprob in API_RESPONSE.choices[0].logprobs.content:\n",
    "        html_output += f'<p style=\"color:cyan\">has_sufficient_context_for_answer: {logprob.token}, <span style=\"color:darkorange\">logprobs: {logprob.logprob}, <span style=\"color:magenta\">linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%</span></p>'\n",
    "\n",
    "display(HTML(html_output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ead8ea0-a9c9-4b7d-b93a-d76cd6c6aa0c",
   "metadata": {},
   "source": [
    "### 3. Highlighter and bytes parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "bba8ced9-0a3e-4860-b22d-8fef532a129b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:04:01.707303Z",
     "iopub.status.busy": "2024-12-06T13:04:01.706638Z",
     "iopub.status.idle": "2024-12-06T13:04:04.321310Z",
     "shell.execute_reply": "2024-12-06T13:04:04.318968Z",
     "shell.execute_reply.started": "2024-12-06T13:04:01.707255Z"
    }
   },
   "outputs": [],
   "source": [
    "PROMPT = \"\"\"What's the longest word in the English language?\"\"\"\n",
    "\n",
    "API_RESPONSE = get_completion(\n",
    "    [{\"role\": \"user\", \"content\": PROMPT}], logprobs=True, top_logprobs=5\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "e1d1bc46-a3db-41f9-a6cf-6d738ff5a5d2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:06:59.244765Z",
     "iopub.status.busy": "2024-12-06T13:06:59.244102Z",
     "iopub.status.idle": "2024-12-06T13:06:59.252830Z",
     "shell.execute_reply": "2024-12-06T13:06:59.250659Z",
     "shell.execute_reply.started": "2024-12-06T13:06:59.244715Z"
    }
   },
   "outputs": [],
   "source": [
    "# pprint(API_RESPONSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "15e82c67-8987-48d8-bc97-580c4f3c5596",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:03:32.948021Z",
     "iopub.status.busy": "2024-12-06T13:03:32.947377Z",
     "iopub.status.idle": "2024-12-06T13:03:36.788249Z",
     "shell.execute_reply": "2024-12-06T13:03:36.785883Z",
     "shell.execute_reply.started": "2024-12-06T13:03:32.947974Z"
    }
   },
   "outputs": [],
   "source": [
    "def highlight_text(api_response):\n",
    "    colors = [\n",
    "        \"#FF00FF\",  # Magenta\n",
    "        \"#008000\",  # Green\n",
    "        \"#FF8C00\",  # Dark Orange\n",
    "        \"#FF0000\",  # Red\n",
    "        \"#0000FF\",  # Blue\n",
    "    ]\n",
    "    tokens = api_response.choices[0].logprobs.content\n",
    "\n",
    "    color_idx = 0  # Initialize color index\n",
    "    html_output = \"\"  # Initialize HTML output\n",
    "    for t in tokens:\n",
    "        token_str = bytes(t.bytes).decode(\"utf-8\")  # Decode bytes to string\n",
    "\n",
    "        # Add colored token to HTML output\n",
    "        html_output += f\"<span style='color: {colors[color_idx]}'>{token_str}</span>\"\n",
    "\n",
    "        # Move to the next color\n",
    "        color_idx = (color_idx + 1) % len(colors)\n",
    "    display(HTML(html_output))  # Display HTML output\n",
    "    print(f\"Total number of tokens: {len(tokens)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "bc938b14-b86a-47ff-ab52-f10ce7c5c337",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:03:47.034439Z",
     "iopub.status.busy": "2024-12-06T13:03:47.033781Z",
     "iopub.status.idle": "2024-12-06T13:03:47.047556Z",
     "shell.execute_reply": "2024-12-06T13:03:47.045333Z",
     "shell.execute_reply.started": "2024-12-06T13:03:47.034391Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style='color: #FF00FF'>The</span><span style='color: #008000'> longest</span><span style='color: #FF8C00'> word</span><span style='color: #FF0000'> in</span><span style='color: #0000FF'> the</span><span style='color: #FF00FF'> English</span><span style='color: #008000'> language</span><span style='color: #FF8C00'> is</span><span style='color: #FF0000'> often</span><span style='color: #0000FF'> considered</span><span style='color: #FF00FF'> to</span><span style='color: #008000'> be</span><span style='color: #FF8C00'> a</span><span style='color: #FF0000'> chemical</span><span style='color: #0000FF'> name</span><span style='color: #FF00FF'> for</span><span style='color: #008000'> the</span><span style='color: #FF8C00'> protein</span><span style='color: #FF0000'> known</span><span style='color: #0000FF'> as</span><span style='color: #FF00FF'> tit</span><span style='color: #008000'>in</span><span style='color: #FF8C00'>,</span><span style='color: #FF0000'> which</span><span style='color: #0000FF'> is</span><span style='color: #FF00FF'> sometimes</span><span style='color: #008000'> referred</span><span style='color: #FF8C00'> to</span><span style='color: #FF0000'> as</span><span style='color: #0000FF'> \"</span><span style='color: #FF00FF'>meth</span><span style='color: #008000'>ion</span><span style='color: #FF8C00'>yl</span><span style='color: #FF0000'>th</span><span style='color: #0000FF'>reon</span><span style='color: #FF00FF'>yl</span><span style='color: #008000'>th</span><span style='color: #FF8C00'>reon</span><span style='color: #FF0000'>yl</span><span style='color: #0000FF'>gl</span><span style='color: #FF00FF'>ut</span><span style='color: #008000'>amin</span><span style='color: #FF8C00'>yl</span><span style='color: #FF0000'>al</span><span style='color: #0000FF'>anyl</span><span style='color: #FF00FF'>pro</span><span style='color: #008000'>ly</span><span style='color: #FF8C00'>l</span><span style='color: #FF0000'>...</span><span style='color: #0000FF'>is</span><span style='color: #FF00FF'>ole</span><span style='color: #008000'>uc</span><span style='color: #FF8C00'>ine</span><span style='color: #FF0000'>.\"</span><span style='color: #0000FF'> This</span><span style='color: #FF00FF'> word</span><span style='color: #008000'> is</span><span style='color: #FF8C00'> extremely</span><span style='color: #FF0000'> long</span><span style='color: #0000FF'>,</span><span style='color: #FF00FF'> with</span><span style='color: #008000'> over</span><span style='color: #FF8C00'> </span><span style='color: #FF0000'>189</span><span style='color: #0000FF'>,</span><span style='color: #FF00FF'>000</span><span style='color: #008000'> letters</span><span style='color: #FF8C00'>,</span><span style='color: #FF0000'> and</span><span style='color: #0000FF'> is</span><span style='color: #FF00FF'> not</span><span style='color: #008000'> practical</span><span style='color: #FF8C00'> for</span><span style='color: #FF0000'> everyday</span><span style='color: #0000FF'> use</span><span style='color: #FF00FF'>.</span><span style='color: #008000'> \n",
       "\n",
       "</span><span style='color: #FF8C00'>In</span><span style='color: #FF0000'> more</span><span style='color: #0000FF'> practical</span><span style='color: #FF00FF'> terms</span><span style='color: #008000'>,</span><span style='color: #FF8C00'> one</span><span style='color: #FF0000'> of</span><span style='color: #0000FF'> the</span><span style='color: #FF00FF'> longest</span><span style='color: #008000'> words</span><span style='color: #FF8C00'> found</span><span style='color: #FF0000'> in</span><span style='color: #0000FF'> major</span><span style='color: #FF00FF'> dictionaries</span><span style='color: #008000'> is</span><span style='color: #FF8C00'> \"</span><span style='color: #FF0000'>p</span><span style='color: #0000FF'>ne</span><span style='color: #FF00FF'>um</span><span style='color: #008000'>on</span><span style='color: #FF8C00'>oul</span><span style='color: #FF0000'>tr</span><span style='color: #0000FF'>amic</span><span style='color: #FF00FF'>ros</span><span style='color: #008000'>cop</span><span style='color: #FF8C00'>ics</span><span style='color: #FF0000'>ilic</span><span style='color: #0000FF'>ovol</span><span style='color: #FF00FF'>can</span><span style='color: #008000'>ocon</span><span style='color: #FF8C00'>iosis</span><span style='color: #FF0000'>,\"</span><span style='color: #0000FF'> a</span><span style='color: #FF00FF'> </span><span style='color: #008000'>45</span><span style='color: #FF8C00'>-letter</span><span style='color: #FF0000'> word</span><span style='color: #0000FF'> referring</span><span style='color: #FF00FF'> to</span><span style='color: #008000'> a</span><span style='color: #FF8C00'> type</span><span style='color: #FF0000'> of</span><span style='color: #0000FF'> lung</span><span style='color: #FF00FF'> disease</span><span style='color: #008000'> caused</span><span style='color: #FF8C00'> by</span><span style='color: #FF0000'> inhal</span><span style='color: #0000FF'>ing</span><span style='color: #FF00FF'> very</span><span style='color: #008000'> fine</span><span style='color: #FF8C00'> sil</span><span style='color: #FF0000'>icate</span><span style='color: #0000FF'> or</span><span style='color: #FF00FF'> quartz</span><span style='color: #008000'> dust</span><span style='color: #FF8C00'>.</span><span style='color: #FF0000'> \n",
       "\n",
       "</span><span style='color: #0000FF'>It's</span><span style='color: #FF00FF'> important</span><span style='color: #008000'> to</span><span style='color: #FF8C00'> note</span><span style='color: #FF0000'> that</span><span style='color: #0000FF'> many</span><span style='color: #FF00FF'> of</span><span style='color: #008000'> these</span><span style='color: #FF8C00'> long</span><span style='color: #FF0000'> words</span><span style='color: #0000FF'> are</span><span style='color: #FF00FF'> technical</span><span style='color: #008000'> or</span><span style='color: #FF8C00'> coined</span><span style='color: #FF0000'> for</span><span style='color: #0000FF'> specific</span><span style='color: #FF00FF'> purposes</span><span style='color: #008000'> and</span><span style='color: #FF8C00'> are</span><span style='color: #FF0000'> not</span><span style='color: #0000FF'> commonly</span><span style='color: #FF00FF'> used</span><span style='color: #008000'> in</span><span style='color: #FF8C00'> everyday</span><span style='color: #FF0000'> language</span><span style='color: #0000FF'>.</span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of tokens: 160\n"
     ]
    }
   ],
   "source": [
    "highlight_text(API_RESPONSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d55d40d-635e-4384-b9fb-3b3c7d08614c",
   "metadata": {},
   "source": [
    "### 4. Autocomplete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2eacac6-4451-43f7-8b7f-2920c2a0279b",
   "metadata": {},
   "source": [
    "- 自动补全，即预测（这也对齐了 GPT 的训练任务，预测下一个单词），预测即推理（reasoning），推理即智能；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2285481d-d315-4938-b4fd-3e4214991efa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:49:17.292167Z",
     "iopub.status.busy": "2024-12-06T12:49:17.291781Z",
     "iopub.status.idle": "2024-12-06T12:49:17.300456Z",
     "shell.execute_reply": "2024-12-06T12:49:17.298206Z",
     "shell.execute_reply.started": "2024-12-06T12:49:17.292128Z"
    }
   },
   "outputs": [],
   "source": [
    "sentence_list = [\n",
    "    \"My\",\n",
    "    \"My least\",\n",
    "    \"My least favorite\",\n",
    "    \"My least favorite TV\",\n",
    "    \"My least favorite TV show\",\n",
    "    \"My least favorite TV show is\",\n",
    "    \"My least favorite TV show is Breaking Bad\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "9a19d2ec-4398-4005-864e-8d90e0326e14",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:05:00.175030Z",
     "iopub.status.busy": "2024-12-06T13:05:00.174712Z",
     "iopub.status.idle": "2024-12-06T13:05:01.430326Z",
     "shell.execute_reply": "2024-12-06T13:05:01.428215Z",
     "shell.execute_reply.started": "2024-12-06T13:05:00.175010Z"
    }
   },
   "outputs": [],
   "source": [
    "PROMPT = \"\"\"Complete this sentence. You are acting as auto-complete. Simply complete the sentence to the best of your ability, make sure it is just ONE sentence: {sentence}\"\"\"\n",
    "API_RESPONSE = get_completion(\n",
    "    [{\"role\": \"user\", \"content\": PROMPT.format(sentence='My least')}],\n",
    "    model=\"gpt-4o-mini\",\n",
    "    logprobs=True,\n",
    "    top_logprobs=3,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "11d5b588-29df-4d3e-82a8-da992f5d173b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:05:08.430895Z",
     "iopub.status.busy": "2024-12-06T13:05:08.430294Z",
     "iopub.status.idle": "2024-12-06T13:05:08.550123Z",
     "shell.execute_reply": "2024-12-06T13:05:08.548100Z",
     "shell.execute_reply.started": "2024-12-06T13:05:08.430833Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletion</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chatcmpl-AbSIuBqXKn7m5VEi5IrGMV5KW0JHm'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">choices</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Choice</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">finish_reason</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'stop'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">index</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprobs</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChoiceLogprobs</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'My'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">77</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.2656334e-05</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'My'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">77</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.2656334e-05</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'favorite'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">118</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.750063</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' My'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">77</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-12.625063</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' least'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' least'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'least'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-18.25</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Least'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">76</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-18.75</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' favorite'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">118</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.8624639e-06</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' favorite'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">118</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.8624639e-06</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' favourite'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">118</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-14.125002</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' preferred'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">112</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-14.750002</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' chore'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">99</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.4413885</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' chore'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">99</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.4413885</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' food'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9413885</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' task'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">107</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6913886</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' is'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0019702658</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' is'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0019702658</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' to'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.3769703</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' has'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.3769703</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' doing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.2812586</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' doing'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.2812586</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' cleaning'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">99</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0312586</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' washing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">119</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.2812586</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' the'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.018234232</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' the'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.018234232</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' laundry'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.0182343</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' dishes'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.393234</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' dishes'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.6326285</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' dishes'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">104</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.6326285</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' laundry'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.7576285</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'lau'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-11.132628</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' after'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.14331494</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' after'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.14331494</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'.'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.5183148</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' because'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">98</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">99</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">117</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.3933148</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' a'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.011266974</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' a'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.011266974</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' dinner'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.511267</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' every'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">118</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.761267</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' long'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.042841256</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' long'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">110</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.042841256</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' big'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">98</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">105</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.2928412</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' large'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">114</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">103</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.4178414</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' day'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0022021208</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' day'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0022021208</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' meal'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">109</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">108</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.502202</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\" day's\"</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">121</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">39</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">115</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.752202</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionTokenLogprob</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'.'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46</span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.00787136</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_logprobs</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'.'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.00787136</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' at'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">97</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">116</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.882871</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   │   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TopLogprob</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">token</span>=<span style=\"color: #008000; text-decoration-color: #008000\">' of'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">bytes</span>=<span style=\"font-weight: bold\">[</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">111</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">102</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">logprob</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.257872</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   │   </span><span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">message</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">ChatCompletionMessage</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">content</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'My least favorite chore is doing the dishes after a long day.'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">refusal</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">role</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'assistant'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">audio</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">function_call</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">tool_calls</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">created</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1733490300</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">model</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'gpt-4o-mini-2024-07-18'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">object</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'chat.completion'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">service_tier</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">system_fingerprint</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'fp_bba3c8e70b'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">usage</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionUsage</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">40</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">total_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">53</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">completion_tokens_details</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">CompletionTokensDetails</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">accepted_prediction_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">audio_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">reasoning_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">rejected_prediction_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">prompt_tokens_details</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">PromptTokensDetails</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">audio_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #808000; text-decoration-color: #808000\">cached_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mChatCompletion\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mid\u001b[0m=\u001b[32m'chatcmpl-AbSIuBqXKn7m5VEi5IrGMV5KW0JHm'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mchoices\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;35mChoice\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mfinish_reason\u001b[0m=\u001b[32m'stop'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mindex\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mlogprobs\u001b[0m=\u001b[1;35mChoiceLogprobs\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'My'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m77\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-6.2656334e-05\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'My'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m77\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-6.2656334e-05\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'favorite'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m102\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m118\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-9.750063\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' My'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m77\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-12.625063\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' least'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m116\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' least'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m116\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.0\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'least'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m108\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m116\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-18.25\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'Least'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m76\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m116\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-18.75\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' favorite'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m102\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m118\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-1.8624639e-06\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' favorite'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m102\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m118\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-1.8624639e-06\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' favourite'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m102\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m118\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m117\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-14.125002\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' preferred'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m112\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m102\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m100\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-14.750002\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' chore'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m99\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.4413885\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' chore'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m99\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.4413885\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' food'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m102\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m100\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-1.9413885\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' task'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m107\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-2.6913886\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' is'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m115\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.0019702658\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' is'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m115\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.0019702658\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' to'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m111\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-7.3769703\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' has'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m115\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-7.3769703\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' doing'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m103\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.2812586\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' doing'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m103\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.2812586\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' cleaning'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m99\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m103\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-2.0312586\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' washing'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m119\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m103\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-2.2812586\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' the'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.018234232\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' the'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.018234232\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' laundry'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m117\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-4.0182343\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' dishes'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m115\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-9.393234\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' dishes'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m115\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.6326285\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' dishes'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m104\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m115\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.6326285\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' laundry'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m117\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.7576285\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'lau'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m108\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m117\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-11.132628\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' after'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m102\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m114\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.14331494\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' after'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m102\u001b[0m, \u001b[1;36m116\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m114\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.14331494\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'.'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m46\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-2.5183148\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' because'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m98\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m99\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m117\u001b[0m, \u001b[1;36m115\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-3.3933148\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' a'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m97\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.011266974\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' a'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m97\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.011266974\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' dinner'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m114\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-4.511267\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' every'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m118\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-8.761267\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' long'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m103\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.042841256\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' long'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m110\u001b[0m, \u001b[1;36m103\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.042841256\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' big'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m98\u001b[0m, \u001b[1;36m105\u001b[0m, \u001b[1;36m103\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-3.2928412\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' large'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m108\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m114\u001b[0m, \u001b[1;36m103\u001b[0m, \u001b[1;36m101\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-5.4178414\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' day'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.0022021208\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' day'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m121\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.0022021208\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' meal'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m109\u001b[0m, \u001b[1;36m101\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m108\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-6.502202\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m\" day's\"\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m100\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m121\u001b[0m, \u001b[1;36m39\u001b[0m, \u001b[1;36m115\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-7.752202\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1;35mChatCompletionTokenLogprob\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'.'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m46\u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.00787136\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[33mtop_logprobs\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m'.'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m46\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-0.00787136\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' at'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m97\u001b[0m, \u001b[1;36m116\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-4.882871\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   │   │   │   \u001b[0m\u001b[1;35mTopLogprob\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtoken\u001b[0m=\u001b[32m' of'\u001b[0m, \u001b[33mbytes\u001b[0m=\u001b[1m[\u001b[0m\u001b[1;36m32\u001b[0m, \u001b[1;36m111\u001b[0m, \u001b[1;36m102\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mlogprob\u001b[0m=\u001b[1;36m-8.257872\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   │   \u001b[0m\u001b[1m]\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mmessage\u001b[0m=\u001b[1;35mChatCompletionMessage\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mcontent\u001b[0m=\u001b[32m'My least favorite chore is doing the dishes after a long day.'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrefusal\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mrole\u001b[0m=\u001b[32m'assistant'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33maudio\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mfunction_call\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   │   \u001b[0m\u001b[33mtool_calls\u001b[0m=\u001b[3;35mNone\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mcreated\u001b[0m=\u001b[1;36m1733490300\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mmodel\u001b[0m=\u001b[32m'gpt-4o-mini-2024-07-18'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mobject\u001b[0m=\u001b[32m'chat.completion'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mservice_tier\u001b[0m=\u001b[3;35mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33msystem_fingerprint\u001b[0m=\u001b[32m'fp_bba3c8e70b'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33musage\u001b[0m=\u001b[1;35mCompletionUsage\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mcompletion_tokens\u001b[0m=\u001b[1;36m13\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mprompt_tokens\u001b[0m=\u001b[1;36m40\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtotal_tokens\u001b[0m=\u001b[1;36m53\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mcompletion_tokens_details\u001b[0m=\u001b[1;35mCompletionTokensDetails\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33maccepted_prediction_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33maudio_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mreasoning_tokens\u001b[0m=\u001b[1;36m0\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[33mrejected_prediction_tokens\u001b[0m=\u001b[1;36m0\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mprompt_tokens_details\u001b[0m=\u001b[1;35mPromptTokensDetails\u001b[0m\u001b[1m(\u001b[0m\u001b[33maudio_tokens\u001b[0m=\u001b[1;36m0\u001b[0m, \u001b[33mcached_tokens\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pprint(API_RESPONSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "043f9618-00c8-4520-9f7f-eba6cb21ffdc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:55:32.393473Z",
     "iopub.status.busy": "2024-12-06T12:55:32.392793Z",
     "iopub.status.idle": "2024-12-06T12:55:32.406104Z",
     "shell.execute_reply": "2024-12-06T12:55:32.404223Z",
     "shell.execute_reply.started": "2024-12-06T12:55:32.393423Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('My least favorite chore is doing the dishes after a long day.',\n",
       " [TopLogprob(token='My', bytes=[77, 121], logprob=-7.934485e-05),\n",
       "  TopLogprob(token='favorite', bytes=[102, 97, 118, 111, 114, 105, 116, 101], logprob=-9.500079),\n",
       "  TopLogprob(token=' My', bytes=[32, 77, 121], logprob=-12.625079)])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "API_RESPONSE.choices[0].message.content, API_RESPONSE.choices[0].logprobs.content[0].top_logprobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "53e3f5b0-0d11-4f99-a5ee-9c66c7c85f37",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T12:51:12.113690Z",
     "iopub.status.busy": "2024-12-06T12:51:12.113020Z",
     "iopub.status.idle": "2024-12-06T12:51:17.971400Z",
     "shell.execute_reply": "2024-12-06T12:51:17.969193Z",
     "shell.execute_reply.started": "2024-12-06T12:51:12.113642Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>Sentence: My</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.00010032469, <span style=\"color:magenta\">linear probability: 99.99%</span></p><p style=\"color:cyan\">Predicted next token: my, <span style=\"color:darkorange\">logprobs: -9.5001, <span style=\"color:magenta\">linear probability: 0.01%</span></p><p style=\"color:cyan\">Predicted next token: favorite, <span style=\"color:darkorange\">logprobs: -11.7501, <span style=\"color:magenta\">linear probability: 0.0%</span></p><br><p>Sentence: My least</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.00014478032, <span style=\"color:magenta\">linear probability: 99.99%</span></p><p style=\"color:cyan\">Predicted next token: favorite, <span style=\"color:darkorange\">logprobs: -8.875145, <span style=\"color:magenta\">linear probability: 0.01%</span></p><p style=\"color:cyan\">Predicted next token:  My, <span style=\"color:darkorange\">logprobs: -12.625145, <span style=\"color:magenta\">linear probability: 0.0%</span></p><br><p>Sentence: My least favorite</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -3.786197e-05, <span style=\"color:magenta\">linear probability: 100.0%</span></p><p style=\"color:cyan\">Predicted next token: my, <span style=\"color:darkorange\">logprobs: -11.375038, <span style=\"color:magenta\">linear probability: 0.0%</span></p><p style=\"color:cyan\">Predicted next token: \"My, <span style=\"color:darkorange\">logprobs: -11.625038, <span style=\"color:magenta\">linear probability: 0.0%</span></p><br><p>Sentence: My least favorite TV</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.10060819, <span style=\"color:magenta\">linear probability: 90.43%</span></p><p style=\"color:cyan\">Predicted next token: show, <span style=\"color:darkorange\">logprobs: -2.350608, <span style=\"color:magenta\">linear probability: 9.53%</span></p><p style=\"color:cyan\">Predicted next token: shows, <span style=\"color:darkorange\">logprobs: -8.850608, <span style=\"color:magenta\">linear probability: 0.01%</span></p><br><p>Sentence: My least favorite TV show</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.006730288, <span style=\"color:magenta\">linear probability: 99.33%</span></p><p style=\"color:cyan\">Predicted next token: is, <span style=\"color:darkorange\">logprobs: -5.00673, <span style=\"color:magenta\">linear probability: 0.67%</span></p><p style=\"color:cyan\">Predicted next token: my, <span style=\"color:darkorange\">logprobs: -11.75673, <span style=\"color:magenta\">linear probability: 0.0%</span></p><br><p>Sentence: My least favorite TV show is</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.00046689616, <span style=\"color:magenta\">linear probability: 99.95%</span></p><p style=\"color:cyan\">Predicted next token: \"My, <span style=\"color:darkorange\">logprobs: -7.750467, <span style=\"color:magenta\">linear probability: 0.04%</span></p><p style=\"color:cyan\">Predicted next token: my, <span style=\"color:darkorange\">logprobs: -10.500467, <span style=\"color:magenta\">linear probability: 0.0%</span></p><br><p>Sentence: My least favorite TV show is Breaking Bad</p><p style=\"color:cyan\">Predicted next token: because, <span style=\"color:darkorange\">logprobs: -0.20152454, <span style=\"color:magenta\">linear probability: 81.75%</span></p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -1.7015245, <span style=\"color:magenta\">linear probability: 18.24%</span></p><p style=\"color:cyan\">Predicted next token: Because, <span style=\"color:darkorange\">logprobs: -9.701525, <span style=\"color:magenta\">linear probability: 0.01%</span></p><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "high_prob_completions = {}\n",
    "low_prob_completions = {}\n",
    "html_output = \"\"\n",
    "\n",
    "for sentence in sentence_list:\n",
    "    PROMPT = \"\"\"Complete this sentence. You are acting as auto-complete. Simply complete the sentence to the best of your ability, make sure it is just ONE sentence: {sentence}\"\"\"\n",
    "    API_RESPONSE = get_completion(\n",
    "        [{\"role\": \"user\", \"content\": PROMPT.format(sentence=sentence)}],\n",
    "        model=\"gpt-4o-mini\",\n",
    "        logprobs=True,\n",
    "        top_logprobs=3,\n",
    "    )\n",
    "    html_output += f'<p>Sentence: {sentence}</p>'\n",
    "    first_token = True\n",
    "    for token in API_RESPONSE.choices[0].logprobs.content[0].top_logprobs:\n",
    "        html_output += f'<p style=\"color:cyan\">Predicted next token: {token.token}, <span style=\"color:darkorange\">logprobs: {token.logprob}, <span style=\"color:magenta\">linear probability: {np.round(np.exp(token.logprob)*100,2)}%</span></p>'\n",
    "        if first_token:\n",
    "            if np.exp(token.logprob) > 0.95:\n",
    "                high_prob_completions[sentence] = token.token\n",
    "            if np.exp(token.logprob) < 0.60:\n",
    "                low_prob_completions[sentence] = token.token\n",
    "        first_token = False\n",
    "    html_output += \"<br>\"\n",
    "\n",
    "display(HTML(html_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "44a6262c-fd77-4585-9fe9-7a24c6313d7b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:00:04.625663Z",
     "iopub.status.busy": "2024-12-06T13:00:04.625027Z",
     "iopub.status.idle": "2024-12-06T13:00:15.649307Z",
     "shell.execute_reply": "2024-12-06T13:00:15.647025Z",
     "shell.execute_reply.started": "2024-12-06T13:00:04.625615Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>Sentence: My</p><p style=\"color:cyan\">Predicted next token: dog, <span style=\"color:darkorange\">logprobs: -0.736721, <span style=\"color:magenta\">linear probability: 47.87%</span></p><p style=\"color:cyan\">Predicted next token: favorite, <span style=\"color:darkorange\">logprobs: -1.3540478, <span style=\"color:magenta\">linear probability: 25.82%</span></p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -1.7375298, <span style=\"color:magenta\">linear probability: 17.6%</span></p><br><p>Sentence: My least</p><p style=\"color:cyan\">Predicted next token: favorite, <span style=\"color:darkorange\">logprobs: -0.0031446815, <span style=\"color:magenta\">linear probability: 99.69%</span></p><p style=\"color:cyan\">Predicted next token: f, <span style=\"color:darkorange\">logprobs: -6.4583488, <span style=\"color:magenta\">linear probability: 0.16%</span></p><p style=\"color:cyan\">Predicted next token: favor, <span style=\"color:darkorange\">logprobs: -6.992248, <span style=\"color:magenta\">linear probability: 0.09%</span></p><br><p>Sentence: My least favorite</p><p style=\"color:cyan\">Predicted next token: thing, <span style=\"color:darkorange\">logprobs: -0.68627626, <span style=\"color:magenta\">linear probability: 50.34%</span></p><p style=\"color:cyan\">Predicted next token: part, <span style=\"color:darkorange\">logprobs: -1.5119522, <span style=\"color:magenta\">linear probability: 22.05%</span></p><p style=\"color:cyan\">Predicted next token: activity, <span style=\"color:darkorange\">logprobs: -2.3348892, <span style=\"color:magenta\">linear probability: 9.68%</span></p><br><p>Sentence: My least favorite TV</p><p style=\"color:cyan\">Predicted next token: show, <span style=\"color:darkorange\">logprobs: -0.0004896457, <span style=\"color:magenta\">linear probability: 99.95%</span></p><p style=\"color:cyan\">Predicted next token: series, <span style=\"color:darkorange\">logprobs: -8.402778, <span style=\"color:magenta\">linear probability: 0.02%</span></p><p style=\"color:cyan\">Predicted next token: shows, <span style=\"color:darkorange\">logprobs: -8.887841, <span style=\"color:magenta\">linear probability: 0.01%</span></p><br><p>Sentence: My least favorite TV show</p><p style=\"color:cyan\">Predicted next token: is, <span style=\"color:darkorange\">logprobs: -0.00061033305, <span style=\"color:magenta\">linear probability: 99.94%</span></p><p style=\"color:cyan\">Predicted next token: has, <span style=\"color:darkorange\">logprobs: -8.137676, <span style=\"color:magenta\">linear probability: 0.03%</span></p><p style=\"color:cyan\">Predicted next token: would, <span style=\"color:darkorange\">logprobs: -9.002552, <span style=\"color:magenta\">linear probability: 0.01%</span></p><br><p>Sentence: My least favorite TV show is</p><p style=\"color:cyan\">Predicted next token: \"The, <span style=\"color:darkorange\">logprobs: -0.3289613, <span style=\"color:magenta\">linear probability: 71.97%</span></p><p style=\"color:cyan\">Predicted next token: \", <span style=\"color:darkorange\">logprobs: -1.9435358, <span style=\"color:magenta\">linear probability: 14.32%</span></p><p style=\"color:cyan\">Predicted next token: The, <span style=\"color:darkorange\">logprobs: -2.584714, <span style=\"color:magenta\">linear probability: 7.54%</span></p><br><p>Sentence: My least favorite TV show is Breaking Bad</p><p style=\"color:cyan\">Predicted next token: because, <span style=\"color:darkorange\">logprobs: -0.001178201, <span style=\"color:magenta\">linear probability: 99.88%</span></p><p style=\"color:cyan\">Predicted next token: not, <span style=\"color:darkorange\">logprobs: -8.056075, <span style=\"color:magenta\">linear probability: 0.03%</span></p><p style=\"color:cyan\">Predicted next token: ,, <span style=\"color:darkorange\">logprobs: -8.292899, <span style=\"color:magenta\">linear probability: 0.03%</span></p><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "high_prob_completions = {}\n",
    "low_prob_completions = {}\n",
    "html_output = \"\"\n",
    "\n",
    "for sentence in sentence_list:\n",
    "    PROMPT = \"\"\"Complete this sentence. You are acting as auto-complete. Simply complete the sentence to the best of your ability, make sure it is just ONE sentence: {sentence}\"\"\"\n",
    "    API_RESPONSE = get_completion(\n",
    "        [{\"role\": \"user\", \"content\": PROMPT.format(sentence=sentence)}],\n",
    "        logprobs=True,\n",
    "        model='gpt-4',\n",
    "        top_logprobs=3,\n",
    "    )\n",
    "    html_output += f'<p>Sentence: {sentence}</p>'\n",
    "    first_token = True\n",
    "    for token in API_RESPONSE.choices[0].logprobs.content[0].top_logprobs:\n",
    "        html_output += f'<p style=\"color:cyan\">Predicted next token: {token.token}, <span style=\"color:darkorange\">logprobs: {token.logprob}, <span style=\"color:magenta\">linear probability: {np.round(np.exp(token.logprob)*100,2)}%</span></p>'\n",
    "        if first_token:\n",
    "            if np.exp(token.logprob) > 0.95:\n",
    "                high_prob_completions[sentence] = token.token\n",
    "            if np.exp(token.logprob) < 0.60:\n",
    "                low_prob_completions[sentence] = token.token\n",
    "        first_token = False\n",
    "    html_output += \"<br>\"\n",
    "\n",
    "display(HTML(html_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "40c44884-5f32-4d0d-971a-c8235f2b8dd3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:00:45.446937Z",
     "iopub.status.busy": "2024-12-06T13:00:45.446349Z",
     "iopub.status.idle": "2024-12-06T13:00:45.458915Z",
     "shell.execute_reply": "2024-12-06T13:00:45.456810Z",
     "shell.execute_reply.started": "2024-12-06T13:00:45.446890Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'My least': 'favorite',\n",
       "  'My least favorite TV': 'show',\n",
       "  'My least favorite TV show': 'is',\n",
       "  'My least favorite TV show is Breaking Bad': 'because'},\n",
       " {'My': 'dog', 'My least favorite': 'thing'})"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_prob_completions, low_prob_completions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "d55cd7f6-b584-4f63-9095-5854c6e42b0a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:01:00.087086Z",
     "iopub.status.busy": "2024-12-06T13:01:00.086430Z",
     "iopub.status.idle": "2024-12-06T13:01:07.458309Z",
     "shell.execute_reply": "2024-12-06T13:01:07.455988Z",
     "shell.execute_reply.started": "2024-12-06T13:01:00.087038Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>Sentence: My</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.04097642, <span style=\"color:magenta\">linear probability: 95.99%</span></p><p style=\"color:cyan\">Predicted next token: ap, <span style=\"color:darkorange\">logprobs: -3.7909765, <span style=\"color:magenta\">linear probability: 2.26%</span></p><p style=\"color:cyan\">Predicted next token: dog, <span style=\"color:darkorange\">logprobs: -4.2909765, <span style=\"color:magenta\">linear probability: 1.37%</span></p><br><p>Sentence: My least</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.20163347, <span style=\"color:magenta\">linear probability: 81.74%</span></p><p style=\"color:cyan\">Predicted next token: favorite, <span style=\"color:darkorange\">logprobs: -1.7016335, <span style=\"color:magenta\">linear probability: 18.24%</span></p><p style=\"color:cyan\">Predicted next token:  My, <span style=\"color:darkorange\">logprobs: -9.201633, <span style=\"color:magenta\">linear probability: 0.01%</span></p><br><p>Sentence: My least favorite</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.07886687, <span style=\"color:magenta\">linear probability: 92.42%</span></p><p style=\"color:cyan\">Predicted next token: food, <span style=\"color:darkorange\">logprobs: -3.828867, <span style=\"color:magenta\">linear probability: 2.17%</span></p><p style=\"color:cyan\">Predicted next token: subject, <span style=\"color:darkorange\">logprobs: -4.328867, <span style=\"color:magenta\">linear probability: 1.32%</span></p><br><p>Sentence: My least favorite TV</p><p style=\"color:cyan\">Predicted next token: show, <span style=\"color:darkorange\">logprobs: -0.57662326, <span style=\"color:magenta\">linear probability: 56.18%</span></p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.82662326, <span style=\"color:magenta\">linear probability: 43.75%</span></p><p style=\"color:cyan\">Predicted next token:  show, <span style=\"color:darkorange\">logprobs: -8.201623, <span style=\"color:magenta\">linear probability: 0.03%</span></p><br><p>Sentence: My least favorite TV show</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.40044966, <span style=\"color:magenta\">linear probability: 67.0%</span></p><p style=\"color:cyan\">Predicted next token: is, <span style=\"color:darkorange\">logprobs: -1.1504496, <span style=\"color:magenta\">linear probability: 31.65%</span></p><p style=\"color:cyan\">Predicted next token: was, <span style=\"color:darkorange\">logprobs: -4.6504498, <span style=\"color:magenta\">linear probability: 0.96%</span></p><br><p>Sentence: My least favorite TV show is</p><p style=\"color:cyan\">Predicted next token: My, <span style=\"color:darkorange\">logprobs: -0.47896808, <span style=\"color:magenta\">linear probability: 61.94%</span></p><p style=\"color:cyan\">Predicted next token: one, <span style=\"color:darkorange\">logprobs: -1.7289681, <span style=\"color:magenta\">linear probability: 17.75%</span></p><p style=\"color:cyan\">Predicted next token: the, <span style=\"color:darkorange\">logprobs: -2.9789681, <span style=\"color:magenta\">linear probability: 5.08%</span></p><br><p>Sentence: My least favorite TV show is Breaking Bad</p><p style=\"color:cyan\">Predicted next token: because, <span style=\"color:darkorange\">logprobs: -0.066972695, <span style=\"color:magenta\">linear probability: 93.52%</span></p><p style=\"color:cyan\">Predicted next token: ,, <span style=\"color:darkorange\">logprobs: -3.3169727, <span style=\"color:magenta\">linear probability: 3.63%</span></p><p style=\"color:cyan\">Predicted next token:  because, <span style=\"color:darkorange\">logprobs: -4.0669727, <span style=\"color:magenta\">linear probability: 1.71%</span></p><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "high_prob_completions = {}\n",
    "low_prob_completions = {}\n",
    "html_output = \"\"\n",
    "\n",
    "for sentence in sentence_list:\n",
    "    PROMPT = \"\"\"Complete this sentence. You are acting as auto-complete. Simply complete the sentence to the best of your ability, make sure it is just ONE sentence: {sentence}\"\"\"\n",
    "    API_RESPONSE = get_completion(\n",
    "        [{\"role\": \"user\", \"content\": PROMPT.format(sentence=sentence)}],\n",
    "        logprobs=True,\n",
    "        top_logprobs=3,\n",
    "    )\n",
    "    html_output += f'<p>Sentence: {sentence}</p>'\n",
    "    first_token = True\n",
    "    for token in API_RESPONSE.choices[0].logprobs.content[0].top_logprobs:\n",
    "        html_output += f'<p style=\"color:cyan\">Predicted next token: {token.token}, <span style=\"color:darkorange\">logprobs: {token.logprob}, <span style=\"color:magenta\">linear probability: {np.round(np.exp(token.logprob)*100,2)}%</span></p>'\n",
    "        if first_token:\n",
    "            if np.exp(token.logprob) > 0.95:\n",
    "                high_prob_completions[sentence] = token.token\n",
    "            if np.exp(token.logprob) < 0.60:\n",
    "                low_prob_completions[sentence] = token.token\n",
    "        first_token = False\n",
    "    html_output += \"<br>\"\n",
    "\n",
    "display(HTML(html_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "e4425474-b26a-4877-8223-78ed2e248869",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:01:09.599901Z",
     "iopub.status.busy": "2024-12-06T13:01:09.599319Z",
     "iopub.status.idle": "2024-12-06T13:01:09.611069Z",
     "shell.execute_reply": "2024-12-06T13:01:09.608878Z",
     "shell.execute_reply.started": "2024-12-06T13:01:09.599855Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'My': 'My'}, {'My least favorite TV': 'show'})"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_prob_completions, low_prob_completions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c14808f-28e0-4068-8dfe-10b27a36ae2a",
   "metadata": {},
   "source": [
    "### 5. Calculating perplexity (PPL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052f4513-e934-4f8a-bb37-c95053acef70",
   "metadata": {},
   "source": [
    "- perplexity, a measure of the uncertainty.\n",
    "- Perplexity can be calculated by exponentiating the negative of the average of the logprobs.\n",
    "    - exp(-avg(logp))\n",
    "- a higher perplexity indicates a more uncertain result, and a lower perplexity indicates a more confident result.\n",
    "- As such, perplexity can be used to both assess the result of an individual model run and also to compare the relative confidence of results between model runs.\n",
    "- While a high confidence doesn't guarantee result accuracy, it can be a helpful signal that can be paired with other evaluation metrics to build a better understanding of your prompt's behavior.\n",
    "\n",
    "$$\n",
    "\\begin{split}\n",
    "PPL&=\\exp\\left(-\\frac1n\\sum_i\\log P(x_i|x_{\\lt i})\\right)\\\\\n",
    "&=\\exp\\left(\\log P(S)^{-\\frac1n}\\right)\\\\\n",
    "&=P(S)^{-\\frac1n}\n",
    "\\end{split}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "47f24007-e350-40ff-a8f4-0896bff101d6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:15:16.820788Z",
     "iopub.status.busy": "2024-12-06T13:15:16.820130Z",
     "iopub.status.idle": "2024-12-06T13:15:18.678163Z",
     "shell.execute_reply": "2024-12-06T13:15:18.675900Z",
     "shell.execute_reply.started": "2024-12-06T13:15:16.820741Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:     In a short sentence, has artifical intelligence grown in the last decade?\n",
      "Response:   Yes, artificial intelligence has grown significantly in the last decade. \n",
      "\n",
      "Tokens:                Yes              ,     artificial   intelligence            has          grown  significantly             in            the           last         decade              .\n",
      "Logprobs:            -0.00          -0.00          -0.00          -0.00          -0.00          -0.24          -0.08          -0.00          -0.00          -0.00          -0.00          -0.00\n",
      "Perplexity: 1.0278720155134078 \n",
      "\n",
      "Prompt:     In a short sentence, what are your thoughts on the future of artificial intelligence?\n",
      "Response:   The future of artificial intelligence holds great potential for transforming industries and improving efficiency, but also raises ethical and societal concerns that must be carefully addressed. \n",
      "\n",
      "Tokens:               The        future            of    artificial  intelligence         holds         great     potential           for  transforming    industries           and     improving    efficiency             ,           but          also        raises       ethical           and      societal      concerns          that          must            be     carefully     addressed             .\n",
      "Logprobs:           -0.18         -0.02         -0.00         -0.00         -0.00         -0.22         -0.53         -0.26         -0.04         -1.45         -0.20         -0.09         -0.24         -0.85         -0.18         -0.00         -0.34         -0.06         -0.47         -0.64         -0.20         -0.22         -0.01         -0.14         -0.00         -0.57         -0.48         -0.00\n",
      "Perplexity: 1.3019616728795984 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompts = [\n",
    "    \"In a short sentence, has artifical intelligence grown in the last decade?\",\n",
    "    \"In a short sentence, what are your thoughts on the future of artificial intelligence?\",\n",
    "]\n",
    "\n",
    "for prompt in prompts:\n",
    "    API_RESPONSE = get_completion(\n",
    "        [{\"role\": \"user\", \"content\": prompt}],\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        logprobs=True,\n",
    "    )\n",
    "\n",
    "    logprobs = [token.logprob for token in API_RESPONSE.choices[0].logprobs.content]\n",
    "    response_text = API_RESPONSE.choices[0].message.content\n",
    "    response_text_tokens = [token.token for token in API_RESPONSE.choices[0].logprobs.content]\n",
    "    max_starter_length = max(len(s) for s in [\"Prompt:\", \"Response:\", \"Tokens:\", \"Logprobs:\", \"Perplexity:\"])\n",
    "    max_token_length = max(len(s) for s in response_text_tokens)\n",
    "    \n",
    "\n",
    "    formatted_response_tokens = [s.rjust(max_token_length) for s in response_text_tokens]\n",
    "    formatted_lps = [f\"{lp:.2f}\".rjust(max_token_length) for lp in logprobs]\n",
    "\n",
    "    perplexity_score = np.exp(-np.mean(logprobs))\n",
    "    print(\"Prompt:\".ljust(max_starter_length), prompt)\n",
    "    print(\"Response:\".ljust(max_starter_length), response_text, \"\\n\")\n",
    "    print(\"Tokens:\".ljust(max_starter_length), \" \".join(formatted_response_tokens))\n",
    "    print(\"Logprobs:\".ljust(max_starter_length), \" \".join(formatted_lps))\n",
    "    print(\"Perplexity:\".ljust(max_starter_length), perplexity_score, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "238b6f20-4f44-488f-a828-2291d21af669",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-12-06T13:15:33.936725Z",
     "iopub.status.busy": "2024-12-06T13:15:33.936067Z",
     "iopub.status.idle": "2024-12-06T13:15:36.186224Z",
     "shell.execute_reply": "2024-12-06T13:15:36.183967Z",
     "shell.execute_reply.started": "2024-12-06T13:15:33.936676Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:     In a short sentence, has artifical intelligence grown in the last decade?\n",
      "Response:   Yes, artificial intelligence has significantly grown in the last decade, advancing in capabilities and applications. \n",
      "\n",
      "Tokens:                Yes              ,     artificial   intelligence            has  significantly          grown             in            the           last         decade              ,      advancing             in   capabilities            and   applications              .\n",
      "Logprobs:            -0.00           0.00          -0.00          -0.00          -0.00          -0.80          -0.28          -0.58          -0.23          -0.00           0.00          -0.08          -0.26          -0.03          -0.40          -0.39          -0.03          -0.69\n",
      "Perplexity: 1.2335266571550456 \n",
      "\n",
      "Prompt:     In a short sentence, what are your thoughts on the future of artificial intelligence?\n",
      "Response:   The future of artificial intelligence holds immense potential for innovation and transformation across various sectors, but it also requires careful consideration of ethical and societal impacts. \n",
      "\n",
      "Tokens:                 The          future              of      artificial    intelligence           holds         immense       potential             for      innovation             and  transformation          across         various         sectors               ,             but              it            also        requires         careful   consideration              of         ethical             and        societal         impacts               .\n",
      "Logprobs:             -0.10           -0.00            0.00           -0.00            0.00           -0.05           -0.39           -0.01           -0.02           -0.76           -0.13           -0.13           -0.19           -0.11           -0.25           -0.01           -0.17           -0.05           -0.07           -0.53           -0.02           -0.35           -0.00           -0.00           -0.45           -0.01           -0.43           -0.00\n",
      "Perplexity: 1.163746216545455 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "prompts = [\n",
    "    \"In a short sentence, has artifical intelligence grown in the last decade?\",\n",
    "    \"In a short sentence, what are your thoughts on the future of artificial intelligence?\",\n",
    "]\n",
    "\n",
    "for prompt in prompts:\n",
    "    API_RESPONSE = get_completion(\n",
    "        [{\"role\": \"user\", \"content\": prompt}],\n",
    "        logprobs=True,\n",
    "    )\n",
    "\n",
    "    logprobs = [token.logprob for token in API_RESPONSE.choices[0].logprobs.content]\n",
    "    response_text = API_RESPONSE.choices[0].message.content\n",
    "    response_text_tokens = [token.token for token in API_RESPONSE.choices[0].logprobs.content]\n",
    "    max_starter_length = max(len(s) for s in [\"Prompt:\", \"Response:\", \"Tokens:\", \"Logprobs:\", \"Perplexity:\"])\n",
    "    max_token_length = max(len(s) for s in response_text_tokens)\n",
    "    \n",
    "\n",
    "    formatted_response_tokens = [s.rjust(max_token_length) for s in response_text_tokens]\n",
    "    formatted_lps = [f\"{lp:.2f}\".rjust(max_token_length) for lp in logprobs]\n",
    "\n",
    "    perplexity_score = np.exp(-np.mean(logprobs))\n",
    "    print(\"Prompt:\".ljust(max_starter_length), prompt)\n",
    "    print(\"Response:\".ljust(max_starter_length), response_text, \"\\n\")\n",
    "    print(\"Tokens:\".ljust(max_starter_length), \" \".join(formatted_response_tokens))\n",
    "    print(\"Logprobs:\".ljust(max_starter_length), \" \".join(formatted_lps))\n",
    "    print(\"Perplexity:\".ljust(max_starter_length), perplexity_score, \"\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
